{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "08_introduction_to_nlp_in_tensorflow_video.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNKOCtVw0yctgwcHu2wrFF+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrdbourke/tensorflow-deep-learning/blob/main/video_notebooks/08_introduction_to_nlp_in_tensorflow_video.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VkhpvcnZiNkb"
      },
      "source": [
        "# Introduction to NLP Fundamentals in TensorFlow\n",
        "\n",
        "NLP has the goal of deriving informaton out of natural language (could be seqeuences text or speech).\n",
        "\n",
        "Another common term for NLP problems is sequence to sequence problems (seq2seq).\n",
        "\n",
        "> 📖 **Resource:** See all course materials, resources and extra-curriculum for this notebook on GitHub: https://github.com/mrdbourke/tensorflow-deep-learning/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F251SjqZimVF"
      },
      "source": [
        "## Check for GPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m1dVFu5Jis9Z",
        "outputId": "6b269000-e2b8-4bee-8d65-447ef3a20986"
      },
      "source": [
        "!nvidia-smi -L"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GPU 0: Tesla K80 (UUID: GPU-a1b19013-e9dc-a380-ac72-f245433843e9)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9qrnNJ_4iuX-"
      },
      "source": [
        "## Get helper functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xC2eRi-Ti6Bt",
        "outputId": "fd95b8a9-8639-472f-dd5e-fd9fc348fb00"
      },
      "source": [
        "!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
        "\n",
        "# Import series of helper functions for the notebook\n",
        "from helper_functions import unzip_data, create_tensorboard_callback, plot_loss_curves, compare_historys"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-14 05:57:54--  https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 9304 (9.1K) [text/plain]\n",
            "Saving to: ‘helper_functions.py’\n",
            "\n",
            "helper_functions.py 100%[===================>]   9.09K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-04-14 05:57:54 (50.0 MB/s) - ‘helper_functions.py’ saved [9304/9304]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x8MQn_DwjSlJ"
      },
      "source": [
        "## Get a text dataset\n",
        "\n",
        "The dataset we're going to be using is Kaggle's introduction to NLP dataset (text samples of Tweets labelled as diaster or not diaster).\n",
        "\n",
        "See the original source here: https://www.kaggle.com/c/nlp-getting-started"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XuFNZ6zvjW-J",
        "outputId": "8c7a1370-f981-4875-ac12-f79338125c1c"
      },
      "source": [
        "!wget https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
        "\n",
        "# Unzip data\n",
        "unzip_data(\"nlp_getting_started.zip\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-14 05:57:56--  https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.140.128, 108.177.15.128, 173.194.76.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.140.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607343 (593K) [application/zip]\n",
            "Saving to: ‘nlp_getting_started.zip’\n",
            "\n",
            "nlp_getting_started 100%[===================>] 593.11K  --.-KB/s    in 0.007s  \n",
            "\n",
            "2021-04-14 05:57:57 (81.0 MB/s) - ‘nlp_getting_started.zip’ saved [607343/607343]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-I5zOcnvj4NO"
      },
      "source": [
        "## Visualizing a text dataset\n",
        "\n",
        "To visualize our text samples, we first have to read them in, one way to do so would be to use Python: https://realpython.com/read-write-files-python/\n",
        "\n",
        "But I prefer to get visual straight away.\n",
        "\n",
        "So another way to do this is to use pandas..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "WMK3xqK8kqQL",
        "outputId": "168211c3-3ea1-4ef0-ec33-0466fc488366"
      },
      "source": [
        "import pandas as pd\n",
        "train_df = pd.read_csv(\"train.csv\")\n",
        "test_df = pd.read_csv(\"test.csv\")\n",
        "train_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword  ...                                               text target\n",
              "0   1     NaN  ...  Our Deeds are the Reason of this #earthquake M...      1\n",
              "1   4     NaN  ...             Forest fire near La Ronge Sask. Canada      1\n",
              "2   5     NaN  ...  All residents asked to 'shelter in place' are ...      1\n",
              "3   6     NaN  ...  13,000 people receive #wildfires evacuation or...      1\n",
              "4   7     NaN  ...  Just got sent this photo from Ruby #Alaska as ...      1\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "ng4p6stElbl4",
        "outputId": "a9640765-8a02-46fd-a3b2-015f47070cff"
      },
      "source": [
        "# Shuffle training dataframe\n",
        "train_df_shuffled = train_df.sample(frac=1, random_state=42) \n",
        "train_df_shuffled.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2644</th>\n",
              "      <td>3796</td>\n",
              "      <td>destruction</td>\n",
              "      <td>NaN</td>\n",
              "      <td>So you have a new weapon that can cause un-ima...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2227</th>\n",
              "      <td>3185</td>\n",
              "      <td>deluge</td>\n",
              "      <td>NaN</td>\n",
              "      <td>The f$&amp;amp;@ing things I do for #GISHWHES Just...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5448</th>\n",
              "      <td>7769</td>\n",
              "      <td>police</td>\n",
              "      <td>UK</td>\n",
              "      <td>DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>191</td>\n",
              "      <td>aftershock</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Aftershock back to school kick off was great. ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6845</th>\n",
              "      <td>9810</td>\n",
              "      <td>trauma</td>\n",
              "      <td>Montgomery County, MD</td>\n",
              "      <td>in response to trauma Children of Addicts deve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id  ... target\n",
              "2644  3796  ...      1\n",
              "2227  3185  ...      0\n",
              "5448  7769  ...      1\n",
              "132    191  ...      0\n",
              "6845  9810  ...      0\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "dBUTSefKljdb",
        "outputId": "6ebbf625-3331-4043-9a64-fc688ca6eed1"
      },
      "source": [
        "# What does the test dataframe look like?\n",
        "test_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword location                                               text\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4rCFdfq7mJGs",
        "outputId": "42bbb29d-d22a-4bcc-8295-f7f098d8e06a"
      },
      "source": [
        "# How many examples of each class?\n",
        "train_df.target.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h5wrCzWCmS9P",
        "outputId": "50cd45e1-4faf-4d7e-827c-7fc00415688f"
      },
      "source": [
        "# How many total samples?\n",
        "len(train_df), len(test_df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(7613, 3263)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IHqct-_WmzSp",
        "outputId": "b7a07218-9cb6-4a52-d129-0de5644064f9"
      },
      "source": [
        "# Let's visualize some random training examples\n",
        "import random\n",
        "random_index = random.randint(0, len(train_df)-5) # create random indexes not higher than the total number of samples\n",
        "for row in train_df_shuffled[[\"text\", \"target\"]][random_index:random_index+5].itertuples():\n",
        "  _, text, target = row\n",
        "  print(f\"Target: {target}\", \"(real diaster)\" if target > 0 else \"(not real diaster)\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"---\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Target: 0 (not real diaster)\n",
            "Text:\n",
            "Do you have a plan in case of a pool chemical emergency? Learn more here: http://t.co/UePPjwvLcb #watersafety @CDC\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real diaster)\n",
            "Text:\n",
            "I crashed my car into a parked car the other day... #modestmouseremix #truestory\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real diaster)\n",
            "Text:\n",
            "smokers that ruin that new car smell ????\n",
            "\n",
            "---\n",
            "\n",
            "Target: 1 (real diaster)\n",
            "Text:\n",
            "#computers #gadgets Two giant cranes holding a bridge collapse into nearby homes http://t.co/UZIWgZRynY #slingnews\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real diaster)\n",
            "Text:\n",
            "I love the sound of thunder rumbling across the mountains.\n",
            "\n",
            "---\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cZEKNKZOn0A8"
      },
      "source": [
        "### Split data into training and validation sets\n",
        "\n",
        "We want to be able to see how our model is performing on unseen data whilst it trains.\n",
        "\n",
        "And because the testing dataset doesn't have labels, we'll have to create a validation dataset to evaluate on (the model won't see the validation dataset during training so we can use its samples and labels to evaluate our model's performance)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RRurDp_XpEv_"
      },
      "source": [
        "from sklearn.model_selection import train_test_split "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kBSr2LTvpRQm"
      },
      "source": [
        "# Use train_test_split to split training data into training and validation sets\n",
        "train_sentences, val_sentences, train_labels, val_labels = train_test_split(train_df_shuffled[\"text\"].to_numpy(),\n",
        "                                                                            train_df_shuffled[\"target\"].to_numpy(),\n",
        "                                                                            test_size=0.1, # use 10% of training data for validation split\n",
        "                                                                            random_state=42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AxpYGpF5pxiM",
        "outputId": "dff90d32-2ec1-4e81-d32d-87a697334383"
      },
      "source": [
        "# Check the lengths\n",
        "len(train_sentences), len(train_labels), len(val_sentences), len(val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(6851, 6851, 762, 762)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m1xC2MWxp5Rj",
        "outputId": "e5421f13-8871-4fc5-87d0-cff426878e99"
      },
      "source": [
        "# Check the first 10 samples \n",
        "train_sentences[:10], train_labels[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "        'Imagine getting flattened by Kurt Zouma',\n",
              "        '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "        \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "        'Somehow find you and I collide http://t.co/Ee8RpOahPk',\n",
              "        '@EvaHanderek @MarleyKnysh great times until the bus driver held us hostage in the mall parking lot lmfao',\n",
              "        'destroy the free fandom honestly',\n",
              "        'Weapons stolen from National Guard Armory in New Albany still missing #Gunsense http://t.co/lKNU8902JE',\n",
              "        '@wfaaweather Pete when will the heat wave pass? Is it really going to be mid month? Frisco Boy Scouts have a canoe trip in Okla.',\n",
              "        'Patient-reported outcomes in long-term survivors of metastatic colorectal cancer - British Journal of Surgery http://t.co/5Yl4DC1Tqt'],\n",
              "       dtype=object), array([0, 0, 1, 0, 0, 1, 1, 0, 1, 1]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Td1h4t5p7ST"
      },
      "source": [
        "## Converting text into numbers\n",
        "\n",
        "When dealing with a text problem, one of the first things you'll have to do before you can build a model is to convert your text to numbers.\n",
        "\n",
        "There are a few ways to do this, namely:\n",
        "* Tokenziation - direct mapping of token (a token could be a word or a character) to number\n",
        "* Embedding - create a matrix of feature vector for each token (the size of the feature vector can be defined and this embedding can be learned)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI3K0U_xBtu6"
      },
      "source": [
        "### Text vectorization (tokenization)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I_hj9h10FJ56",
        "outputId": "ef9dd7f7-2f60-4d34-8fc4-57ceb83ffa51"
      },
      "source": [
        "train_sentences[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "       'Imagine getting flattened by Kurt Zouma',\n",
              "       '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "       \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "       'Somehow find you and I collide http://t.co/Ee8RpOahPk'],\n",
              "      dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "blJWemjwFIP-"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "# Use the default TextVectorization parameters\n",
        "text_vectorizer = TextVectorization(max_tokens=None, # how many words in the vocabulary (automatically add <OOV>)\n",
        "                                    standardize=\"lower_and_strip_punctuation\",\n",
        "                                    split=\"whitespace\",\n",
        "                                    ngrams=None, # create groups of n-words?\n",
        "                                    output_mode=\"int\", # how to map tokens to numbers\n",
        "                                    output_sequence_length=None, # how long do you want your sequences to be?\n",
        "                                    pad_to_max_tokens=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_wxF2xv-H4xB",
        "outputId": "f84230f1-69fe-4c34-f3ef-bc2784d8ab2f"
      },
      "source": [
        "len(train_sentences[0].split())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "7"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YjPouAI-FR0c",
        "outputId": "2269c697-5b8d-446b-b256-5a6ff675600b"
      },
      "source": [
        "# Find the average number of tokens (words) in the training tweets\n",
        "round(sum([len(i.split()) for i in train_sentences])/len(train_sentences))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "15"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fPPKoQ2uIBuY"
      },
      "source": [
        "# Setup text vectorization variables\n",
        "max_vocab_length = 10000 # max number of words to have in our vocabulary\n",
        "max_length = 15 # max length our sequences will be (e.g. how many words from a Tweet does a model see?)\n",
        "\n",
        "text_vectorizer = TextVectorization(max_tokens=max_vocab_length,\n",
        "                                    output_mode=\"int\",\n",
        "                                    output_sequence_length=max_length)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s5TD-rndImmo"
      },
      "source": [
        "# Fit the text vectorizer instance to the training data using the adapt() method\n",
        "text_vectorizer.adapt(train_sentences)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R-ETwZ92JiOx",
        "outputId": "98a580db-6f26-4275-b39f-ca09e8084e26"
      },
      "source": [
        "# Create a sample sentence and tokenize it\n",
        "sample_sentence = \"There's a flood in my street!\"\n",
        "text_vectorizer([sample_sentence])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[264,   3, 232,   4,  13, 698,   0,   0,   0,   0,   0,   0,   0,\n",
              "          0,   0]])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CtG-XWMHJ9Zi",
        "outputId": "5723dbe6-c5fa-4be7-d3b1-930138530818"
      },
      "source": [
        "# Choose a random sentence from the training dataset and tokenize it\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text:\\n{random_sentence}\\\n",
        "        \\n\\nVectorized version:\")\n",
        "text_vectorizer([random_sentence])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original text:\n",
            "A Rocket To The Moon  ? Sleeping With Sirens ?A Rocket To The Moon ????????????        \n",
            "\n",
            "Vectorized version:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[   3, 2732,    5,    2, 2038, 1365,   14,  568,    3, 2732,    5,\n",
              "           2, 2038,    0,    0]])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "np2kAsmlKk13",
        "outputId": "dbf73060-1cdd-465a-ddad-f6f618a75c34"
      },
      "source": [
        "# Get the unique words in the vocabulary\n",
        "words_in_vocab = text_vectorizer.get_vocabulary()\n",
        "top_5_words = words_in_vocab[:5] # the most common words in the vocab\n",
        "bottom_5_words = words_in_vocab[-5:] # the least common words in the vocab\n",
        "print(f\"Most common words in vocab: {top_5_words}\")\n",
        "print(f\"Least common words in vocab: {bottom_5_words}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Most common words in vocab: ['', '[UNK]', 'the', 'a', 'in']\n",
            "Least common words in vocab: ['pages', 'paeds', 'pads', 'padres', 'paddytomlinson1']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w2Kk1wTIK8CV",
        "outputId": "0fa8489c-c27b-43d1-dff8-2ace2d6f03d6"
      },
      "source": [
        "# how long is our vocab?\n",
        "len(words_in_vocab)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rn3N5uRPLAvm"
      },
      "source": [
        "## Creating an Embedding using an Embedding Layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FLdTzqnVe_2i",
        "outputId": "075cdba4-ff3b-4e28-a97b-ce5364fbe006"
      },
      "source": [
        "max_length,"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(15,)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ty5Y34oQMPi8",
        "outputId": "96ff99e2-310f-4c59-94eb-fc7360919119"
      },
      "source": [
        "from tensorflow.keras import layers \n",
        "\n",
        "embedding = layers.Embedding(input_dim=max_vocab_length, # set the input shape\n",
        "                             output_dim=128, # set the size of the embedding vector\n",
        "                             embeddings_initializer=\"uniform\", # default, initialize embedding vectors randomly\n",
        "                             input_length=max_length # how long is each input\n",
        "                             )\n",
        "\n",
        "embedding"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.layers.embeddings.Embedding at 0x7feaf0574dd0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ThScOvPANZzB",
        "outputId": "1fa3fc2d-87f9-472f-a8c6-d73de4004ceb"
      },
      "source": [
        "# Get a random sentence from training set\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text:\\n{random_sentence}\\\n",
        "        \\n\\nEmbedded version:\")\n",
        "\n",
        "# Embed the random sentence (turn it into numerical representation, aka tokenization first)\n",
        "sample_embed = embedding(text_vectorizer([random_sentence]))\n",
        "sample_embed"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original text:\n",
            "Tram travellers evacuated after powerlines come down in Surfers http://t.co/Qsheu3yF0W        \n",
            "\n",
            "Embedded version:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              "array([[[-0.04958296, -0.04139192, -0.04169388, ...,  0.01408607,\n",
              "         -0.01752247,  0.04459331],\n",
              "        [-0.03577176, -0.04699584, -0.02933953, ...,  0.03718685,\n",
              "         -0.02223238,  0.03446043],\n",
              "        [ 0.00107155, -0.00010961, -0.03292809, ...,  0.00652714,\n",
              "          0.02232416,  0.01176478],\n",
              "        ...,\n",
              "        [-0.03103235, -0.03435149, -0.03189705, ...,  0.02484849,\n",
              "          0.03722688, -0.04460208],\n",
              "        [-0.03103235, -0.03435149, -0.03189705, ...,  0.02484849,\n",
              "          0.03722688, -0.04460208],\n",
              "        [-0.03103235, -0.03435149, -0.03189705, ...,  0.02484849,\n",
              "          0.03722688, -0.04460208]]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QplJbX_3N9JH",
        "outputId": "10a44e5c-4de3-47ed-ba62-c55a057b7005"
      },
      "source": [
        "sample_embed[0][0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(128,), dtype=float32, numpy=\n",
              "array([-0.04958296, -0.04139192, -0.04169388,  0.00415834, -0.03827494,\n",
              "        0.04158347,  0.0019039 ,  0.00871357, -0.00932516, -0.01586594,\n",
              "        0.01528205,  0.04454393, -0.04468231, -0.00108589,  0.00225253,\n",
              "        0.01448091, -0.03179061, -0.00050277, -0.04112818, -0.00091634,\n",
              "        0.03626807,  0.00411427, -0.02153314,  0.04078237,  0.03757713,\n",
              "        0.01344851, -0.03662326, -0.01398536, -0.03565912, -0.03843249,\n",
              "       -0.0358387 ,  0.00736227, -0.00768409,  0.00970813, -0.02738259,\n",
              "       -0.00180788, -0.03767512,  0.00508625,  0.03526122, -0.00299063,\n",
              "        0.00639887,  0.0025234 ,  0.01348929, -0.0276189 ,  0.01180274,\n",
              "        0.03836086,  0.00347748, -0.02545147,  0.00669122,  0.04385629,\n",
              "       -0.00325032,  0.03581819,  0.03069656,  0.03818032,  0.04515779,\n",
              "       -0.0241433 , -0.02243618,  0.00921572,  0.027369  , -0.00890359,\n",
              "        0.03530747, -0.01672858,  0.02132766, -0.04820118, -0.01790066,\n",
              "        0.00465311,  0.02284848, -0.01204724,  0.03491762, -0.04205519,\n",
              "       -0.01458354,  0.00791126, -0.02048448,  0.03228721, -0.03667208,\n",
              "        0.00106823,  0.03324391,  0.0422538 ,  0.02891585, -0.03460705,\n",
              "        0.00177984,  0.02835157, -0.0120228 ,  0.04061544, -0.03556901,\n",
              "       -0.00867062, -0.04836357,  0.01907701, -0.04614526,  0.00350878,\n",
              "        0.03557063, -0.0195251 , -0.01875912,  0.04794297, -0.02299914,\n",
              "       -0.03522237,  0.00921373,  0.04920882,  0.01707551,  0.00311904,\n",
              "        0.04674206,  0.02873522,  0.02245071,  0.04599893,  0.00197817,\n",
              "        0.02397777, -0.04227146,  0.01175212,  0.03409339, -0.02918719,\n",
              "        0.02495323, -0.00010234,  0.03023913,  0.02844686,  0.00422872,\n",
              "       -0.00774982, -0.02149474, -0.02379751,  0.00324794,  0.0328068 ,\n",
              "        0.02928049, -0.01885318,  0.0457284 , -0.04480609, -0.01865388,\n",
              "        0.01408607, -0.01752247,  0.04459331], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UN7r1WeFOldx"
      },
      "source": [
        "## Modelling a text dataset (setting up our modelling experiments)\n",
        "\n",
        "Now we've got our data in numerical format, let's start building and comparing different models.\n",
        "\n",
        "* Model 0: Naive Bayes (baseline) - got this from here: https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html\n",
        "* Model 1: Feed-forward neural network (dense model)\n",
        "* Model 2: LSTM model\n",
        "* Model 3: GRU model\n",
        "* Model 4: Bidirectional LSTM\n",
        "* Model 5: 1D Convolutional Neural Network\n",
        "* Model 6: TensorFlow Hub Pretrained Word Embedding (feature extractor)\n",
        "* Model 7: Same as model 6 but using 10% of data\n",
        "\n",
        "For each of these models, we're going to be following the TensorFlow steps in modelling:\n",
        "* Construct the model\n",
        "* Train the model\n",
        "* Make predictions with the model\n",
        "* Track prediction evaluation metrics for later comparison"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XEPVZ04JRrm1",
        "outputId": "05735c78-ebc5-44ed-a8c1-5307f09fc82a"
      },
      "source": [
        "%%time\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Create tokenization and modelling pipeline\n",
        "model_0 = Pipeline([\n",
        "                    (\"tfidf\", TfidfVectorizer()), # convert words to numbers using tfidf - https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html\n",
        "                    (\"clf\", MultinomialNB()) # model the text converted to numbers\n",
        "])\n",
        "\n",
        "# Fit the pipeline to the training data\n",
        "model_0.fit(train_sentences, train_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 160 ms, sys: 4.15 ms, total: 164 ms\n",
            "Wall time: 181 ms\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MEaWSAriT4tp",
        "outputId": "0604501f-70bb-4d43-f244-5040453f479b"
      },
      "source": [
        "# Let's evalaute our baseline model\n",
        "baseline_score = model_0.score(val_sentences, val_labels)\n",
        "print(f\"Our baseline model achieves an accuracy of: {baseline_score*100:.2f}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Our baseline model achieves an accuracy of: 79.27%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dpnyMrKeT8G_",
        "outputId": "85c0331b-b82a-4b13-a36e-b6b46020cfb6"
      },
      "source": [
        "# Make predictions\n",
        "baseline_preds = model_0.predict(val_sentences)\n",
        "baseline_preds[:20]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "20W1OeuLUrYn"
      },
      "source": [
        "### Creating an evaluation function for our model experiments\n",
        "\n",
        "Let's make a function to evaluate our modelling experiment predictions using: \n",
        "* Accuracy\n",
        "* Precision\n",
        "* Recall\n",
        "* F1-score"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E19vhmhtWsDz"
      },
      "source": [
        "# Function to evalaute: accuracy, precision, recall, F1-score\n",
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "\n",
        "def calculate_results(y_true, y_pred):\n",
        "  \"\"\"\n",
        "  Calculates model accuracy, precision, recall and f1 score of a binary classification model.\n",
        "\n",
        "  Args:\n",
        "  ----\n",
        "  y_true = true labels in the form of a 1D array\n",
        "  y_pred = predicted label in the form of a 1D array\n",
        "\n",
        "  Returns a dictionary of accuracy, precision, recall and f1-score between y_true and y_pred.\n",
        "  \"\"\"\n",
        "  # Calculate model accuracy\n",
        "  model_accuracy = accuracy_score(y_true, y_pred) * 100 # get accuracy score in percentage value\n",
        "  # Calculate model precision, recall and f1 score using \"weighted\" avergage\n",
        "  model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_pred, average=\"weighted\")\n",
        "  # Create a dictionary of model results\n",
        "  model_results = {\"accuracy\": model_accuracy,\n",
        "                   \"precision\": model_precision,\n",
        "                   \"recall\": model_recall,\n",
        "                   \"f1\": model_f1}\n",
        "  return model_results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uo42vAnvYNX4",
        "outputId": "c73a4556-5891-4bc6-bedd-6113817e8082"
      },
      "source": [
        "# Get baseline results\n",
        "baseline_results = calculate_results(y_true=val_labels,\n",
        "                                     y_pred=baseline_preds)\n",
        "baseline_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'f1': 0.7862189758049549,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IT_MFZjLYUEN"
      },
      "source": [
        "### Model 1: A simple dense model (feed-forward neural network)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dSC_uEPjZSX-"
      },
      "source": [
        "# Create tensorboard callback (need to create a new one for each model)\n",
        "from helper_functions import create_tensorboard_callback\n",
        "\n",
        "# Create directory to save TensorBoard logs\n",
        "SAVE_DIR = \"model_logs\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xu2-xPY_Z_3I",
        "outputId": "03545174-5022-4847-c5dc-9087c347ccf6"
      },
      "source": [
        "train_sentences[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "       'Imagine getting flattened by Kurt Zouma',\n",
              "       '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "       \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "       'Somehow find you and I collide http://t.co/Ee8RpOahPk'],\n",
              "      dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "53o_tpRMZhXG"
      },
      "source": [
        "# Build model with the Functional API\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=tf.string) # inputs are 1-dimensional strings\n",
        "x = text_vectorizer(inputs) # turn the input text into numbers \n",
        "x = embedding(x)\n",
        "x = layers.GlobalAveragePooling1D()(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_1 = tf.keras.Model(inputs, outputs, name=\"model_1_dense\") # construct the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1K5tE6dia6Ls",
        "outputId": "3caa5739-91fd-434d-e29c-76290260e523"
      },
      "source": [
        "# Compile model\n",
        "model_1.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Fit the model\n",
        "history_1 = model_1.fit(train_sentences,\n",
        "                        train_labels, \n",
        "                        epochs=5,\n",
        "                        validation_data=(val_sentences, val_labels),\n",
        "                        callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,\n",
        "                                                               experiment_name=\"model_1_dense\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_1_dense/20210414-055804\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 8s 23ms/step - loss: 0.6524 - accuracy: 0.6514 - val_loss: 0.5368 - val_accuracy: 0.7533\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.4596 - accuracy: 0.8153 - val_loss: 0.4726 - val_accuracy: 0.7808\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 4s 17ms/step - loss: 0.3493 - accuracy: 0.8604 - val_loss: 0.4570 - val_accuracy: 0.7927\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 4s 17ms/step - loss: 0.2819 - accuracy: 0.8959 - val_loss: 0.4699 - val_accuracy: 0.7992\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 4s 17ms/step - loss: 0.2366 - accuracy: 0.9112 - val_loss: 0.4819 - val_accuracy: 0.7913\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TJr9cNmEbjf-",
        "outputId": "b127120e-af5e-4599-f2aa-a83d73617bbb"
      },
      "source": [
        "# Evaluate model 1\n",
        "model_1.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "24/24 [==============================] - 0s 4ms/step - loss: 0.4819 - accuracy: 0.7913\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.48192545771598816, 0.7913385629653931]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IXoHTxElcp88",
        "outputId": "235f43d4-622b-4d9b-f2fd-d92518f41499"
      },
      "source": [
        "# Make predictions with model_1\n",
        "model_1_pred_probs = model_1.predict(val_sentences)\n",
        "model_1_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.35383463],\n",
              "       [0.81071967],\n",
              "       [0.99782556],\n",
              "       [0.10595703],\n",
              "       [0.08765673],\n",
              "       [0.93484205],\n",
              "       [0.9048694 ],\n",
              "       [0.99414164],\n",
              "       [0.9482436 ],\n",
              "       [0.2645767 ]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "le7bc1SWc2Ga",
        "outputId": "40f29b5f-76aa-4396-d0d9-bf9fdc9a768e"
      },
      "source": [
        "# Convert model_1 pred probs from probabilities to prediction labels\n",
        "model_1_preds = tf.squeeze(tf.round(model_1_pred_probs))\n",
        "model_1_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cAwelDTHdCqb",
        "outputId": "84cca678-7a9b-4536-c64f-6e3c1c54d2cf"
      },
      "source": [
        "val_labels[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 1, 1, 1, 1, 1, 1, 1, 0])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZEk3d_4Rcbs-",
        "outputId": "c7a31267-4761-4541-ebe1-ee803304e23e"
      },
      "source": [
        "# Evaluate model_1 with our evaluation function\n",
        "model_1_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_1_preds)\n",
        "model_1_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.13385826771653,\n",
              " 'f1': 0.7880356865491198,\n",
              " 'precision': 0.7976012509315953,\n",
              " 'recall': 0.7913385826771654}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eG6GQuq0d3MH",
        "outputId": "a160607d-c165-414a-ecc7-20efd3155415"
      },
      "source": [
        "baseline_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'f1': 0.7862189758049549,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y4_Erx95azN5",
        "outputId": "6520aef1-956d-477c-d693-7107df6b1109"
      },
      "source": [
        "import numpy as np\n",
        "np.array(list(model_1_results.values())) > np.array(list(baseline_results.values()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([False, False, False,  True])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rJ51vV3qdzDF"
      },
      "source": [
        "### Visualizing learned embeddings\n",
        "\n",
        "To further understand word embeddings, let's visualize them, to do so, we'll get the weights matrix (embedding matrix) from our embedding layer and visualize it using the Embedding project tool, see the TensorFlow guide for more: https://www.tensorflow.org/tutorials/text/word_embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-JEsoRe2mgY2",
        "outputId": "24fac085-10d1-405c-dd21-f5d29d7674b4"
      },
      "source": [
        "# Get the vocabulary from the text vectorization layer\n",
        "words_in_vocab = text_vectorizer.get_vocabulary()\n",
        "len(words_in_vocab), words_in_vocab[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, ['', '[UNK]', 'the', 'a', 'in', 'to', 'of', 'and', 'i', 'is'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gWj7h27bmrKN",
        "outputId": "0d4c369a-ee81-4e50-c00a-6901bb434e63"
      },
      "source": [
        "model_1.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "elGf0lzDmxhG",
        "outputId": "133c39ed-5ffa-40d4-e523-8c11488b9b13"
      },
      "source": [
        "# Get the weight matrix of embedding layer\n",
        "# (the weights are the numerical patterns between the text in the training dataset that the model has learned)\n",
        "embed_weights = model_1.get_layer(\"embedding\").get_weights()[0]\n",
        "print(embed_weights.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10000, 128)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQosT7XxnLOA"
      },
      "source": [
        "import io\n",
        "\n",
        "# Code to save trained embeddings to file - we got this from here: https://www.tensorflow.org/tutorials/text/word_embeddings#retrieve_the_trained_word_embeddings_and_save_them_to_disk\n",
        "out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n",
        "\n",
        "for index, word in enumerate(words_in_vocab):\n",
        "  if index == 0:\n",
        "    continue  # skip 0, it's padding.\n",
        "  vec = embed_weights[index]\n",
        "  out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
        "  out_m.write(word + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "xhLv8IRApRrh",
        "outputId": "92c3fdbc-5baa-4655-8140-a8daf7229b6b"
      },
      "source": [
        "# Let's download the saved embeddings locally\n",
        "try:\n",
        "  from google.colab import files\n",
        "  files.download('vectors.tsv')\n",
        "  files.download('metadata.tsv')\n",
        "except Exception:\n",
        "  pass"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_b159ede0-f35c-4ce6-a492-cd673ff3bff1\", \"vectors.tsv\", 15379587)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_4a44dbb6-cae2-42de-8995-230a6566f9d8\", \"metadata.tsv\", 80388)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tb0CFUDlqm9w"
      },
      "source": [
        "Our visual word embeddings might not look like much to us, but they help our model understand the relationships between words.\n",
        "\n",
        "For more on a popular type of word embedding and more visual explanations check out the illustrated word2vec: https://jalammar.github.io/illustrated-word2vec/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIGssmQIpqTr"
      },
      "source": [
        "## Model 2: LSTM\n",
        "\n",
        "* LSTM = long short-term memory\n",
        "\n",
        "For more on RNN's/resources to learn, see here: https://github.com/mrdbourke/tensorflow-deep-learning/blob/main/08_introduction_to_nlp_in_tensorflow.ipynb \n",
        "\n",
        "We're going to focus on writing the code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WoEAithmrUjh",
        "outputId": "eb4339fb-9bc5-44e2-e53c-3968c82efa5f"
      },
      "source": [
        "# Create LSTM model\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "print(f\"After embedding: {x.shape}\")\n",
        "# x = layers.LSTM(64, activation=\"tanh\", return_sequences=True)(x) # use return_sequences=True if you want to stack recurrent layers \n",
        "# print(f\"After LSTM cell with return_sequences=True: {x.shape}\")\n",
        "x = layers.LSTM(64, activation=\"tanh\")(x)\n",
        "print(f\"After LSTM cell: {x.shape}\")\n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer to have on top of LSTM layer\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_2 = tf.keras.Model(inputs, outputs, name=\"model_2_LSTM\")\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "After embedding: (None, 15, 128)\n",
            "After LSTM cell: (None, 64)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "COCCOInqtm3r",
        "outputId": "3be56ba5-4f4d-4926-d9b2-8c07535144c6"
      },
      "source": [
        "# Compile model\n",
        "model_2.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Fit model\n",
        "history_2 = model_2.fit(train_sentences,\n",
        "                        train_labels,\n",
        "                        epochs=5,\n",
        "                        validation_data=(val_sentences, val_labels),\n",
        "                        callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,\n",
        "                                                               experiment_name=\"model_2_LSTM\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_2_LSTM/20210414-055829\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 34s 27ms/step - loss: 0.2989 - accuracy: 0.8959 - val_loss: 0.5377 - val_accuracy: 0.7835\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 5s 21ms/step - loss: 0.1550 - accuracy: 0.9440 - val_loss: 0.6231 - val_accuracy: 0.7861\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 5s 23ms/step - loss: 0.1208 - accuracy: 0.9579 - val_loss: 0.7128 - val_accuracy: 0.7782\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.1029 - accuracy: 0.9605 - val_loss: 0.8816 - val_accuracy: 0.7848\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.0790 - accuracy: 0.9713 - val_loss: 0.9632 - val_accuracy: 0.7822\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O0jw2e3Dukbz",
        "outputId": "f1a4975c-14f5-454d-8f19-7fe1da95ce4e"
      },
      "source": [
        "# Make predictions on the validation dataset\n",
        "model_2_pred_probs = model_2.predict(val_sentences)\n",
        "model_2_pred_probs.shape, model_2_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((762, 1), array([[5.2974503e-03],\n",
              "        [7.5034630e-01],\n",
              "        [9.9962139e-01],\n",
              "        [6.2518336e-02],\n",
              "        [3.7986445e-04],\n",
              "        [9.9857640e-01],\n",
              "        [7.8368771e-01],\n",
              "        [9.9976081e-01],\n",
              "        [9.9954385e-01],\n",
              "        [6.7327970e-01]], dtype=float32))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vQFF0dbQu76A",
        "outputId": "21c01f99-2b32-4202-a1d2-2d14f241d25c"
      },
      "source": [
        "# Round out pred probs and reduce to 1-dimensional array\n",
        "model_2_preds = tf.squeeze(tf.round(model_2_pred_probs))\n",
        "model_2_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "23g9TGPFu_2E",
        "outputId": "dec57c07-3986-44f3-dc84-0a10c495ca76"
      },
      "source": [
        "# Calculate our LSTM model's results\n",
        "model_2_preds = tf.squeeze(tf.round(model_2_preds))\n",
        "model_2_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ANaN1vIkvRUI",
        "outputId": "b88165ed-a60f-4763-d72c-8ca2af8e9bce"
      },
      "source": [
        "# Calculate our model 2 results\n",
        "model_2_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_2_preds)\n",
        "model_2_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.21522309711287,\n",
              " 'f1': 0.7801766074787211,\n",
              " 'precision': 0.7840694582051416,\n",
              " 'recall': 0.7821522309711286}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyZecZmtvYfS"
      },
      "source": [
        "## Model 3: GRU\n",
        "\n",
        "GRU = Gated recurrent unit (one of the most popular and useful recurrent layer types)\n",
        "\n",
        "📖 **Resource:** If you want to see the formula for Tanh, a great extension would be to replicate the function in pure TensorFlow, see here: https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html#tanh"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XpHDZdF6TqmN"
      },
      "source": [
        "# Build an RNN using the GRU cell\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.GRU(64, activation=\"tanh\", return_sequences=True)(x) # return_sequences=True is required for stacking recurrent cells\n",
        "# print(x.shape)\n",
        "x = layers.GRU(64, activation=\"tanh\")(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_3 = tf.keras.Model(inputs, outputs, name=\"model_3_GRU\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G5m4ts3vUhDs",
        "outputId": "0521fcb9-d8ce-434b-cc2c-0f62b91e227e"
      },
      "source": [
        "# Get a summary\n",
        "model_3.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_3_GRU\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_3 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "gru (GRU)                    (None, 64)                37248     \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 1,317,313\n",
            "Trainable params: 1,317,313\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VBert-9mVK-3",
        "outputId": "1c0313dd-6d8d-4237-d0f9-30bcd2683e2b"
      },
      "source": [
        "# Compile model_3 (GRU)\n",
        "model_3.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Fit model\n",
        "model_3_history = model_3.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,\n",
        "                                                                     experiment_name=\"model_3_GRU\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_3_GRU/20210414-055924\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 8s 27ms/step - loss: 0.2703 - accuracy: 0.8921 - val_loss: 0.8344 - val_accuracy: 0.7730\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.0711 - accuracy: 0.9745 - val_loss: 0.8041 - val_accuracy: 0.7782\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 5s 21ms/step - loss: 0.0677 - accuracy: 0.9724 - val_loss: 1.0618 - val_accuracy: 0.7717\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 4s 21ms/step - loss: 0.0545 - accuracy: 0.9773 - val_loss: 1.2175 - val_accuracy: 0.7703\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 5s 21ms/step - loss: 0.0532 - accuracy: 0.9783 - val_loss: 1.2406 - val_accuracy: 0.7625\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hOpajuC1VsoA",
        "outputId": "6604997a-6172-4dc3-9491-b889a5f69cdb"
      },
      "source": [
        "# Make predictions with the trained model\n",
        "model_3_pred_probs = model_3.predict(val_sentences)\n",
        "model_3_pred_probs[:10], model_3_pred_probs.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[2.3852123e-04],\n",
              "        [7.9211944e-01],\n",
              "        [9.9979848e-01],\n",
              "        [5.0801184e-02],\n",
              "        [6.2251114e-05],\n",
              "        [9.9927169e-01],\n",
              "        [3.2195503e-01],\n",
              "        [9.9991918e-01],\n",
              "        [9.9979550e-01],\n",
              "        [6.5830976e-01]], dtype=float32), (762, 1))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gWOxXpgbV8gn",
        "outputId": "7f8b4d4e-f1b6-4378-f594-cf984aa81a26"
      },
      "source": [
        "# Convert model 3 pred probs into labels\n",
        "model_3_preds = tf.squeeze(tf.round(model_3_pred_probs))\n",
        "model_3_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 0., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rxJc3qxCWKFk",
        "outputId": "f1fd6b88-acb1-4b89-ad8f-e5bf1d6921c9"
      },
      "source": [
        "# Calculate results for model 3\n",
        "model_3_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_3_preds)\n",
        "model_3_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 76.24671916010499,\n",
              " 'f1': 0.75760506518967,\n",
              " 'precision': 0.7701495378408689,\n",
              " 'recall': 0.7624671916010499}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmE3Nfc9WLZE"
      },
      "source": [
        "## Model 4: Bidirectional RNN model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yIEUGLncW9_n"
      },
      "source": [
        "# Build a bidirectional RNN in TensorFlow\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x) # return_sequences=True required for stacking RNN layers\n",
        "x = layers.Bidirectional(layers.LSTM(64))(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_4 = tf.keras.Model(inputs, outputs, name=\"model_4_bidirectional\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k24AosTtYoqT",
        "outputId": "2c2d8b7e-8c68-4583-af03-77446e171cf2"
      },
      "source": [
        "model_4.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_4_bidirectional\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_4 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "bidirectional (Bidirectional (None, 128)               98816     \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,378,945\n",
            "Trainable params: 1,378,945\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DJTex7vKYuSH",
        "outputId": "2cf6ef8a-34b6-40c3-c5d2-13f8141bd44f"
      },
      "source": [
        "# Compile model\n",
        "model_4.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Fit model\n",
        "history_model_4 = model_4.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,\n",
        "                                                                     experiment_name=\"model_4_bidirectional\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_4_bidirectional/20210414-055952\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 11s 33ms/step - loss: 0.1773 - accuracy: 0.9660 - val_loss: 0.9334 - val_accuracy: 0.7703\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 6s 26ms/step - loss: 0.0481 - accuracy: 0.9788 - val_loss: 1.1132 - val_accuracy: 0.7677\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 5s 25ms/step - loss: 0.0458 - accuracy: 0.9785 - val_loss: 1.2475 - val_accuracy: 0.7743\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 5s 25ms/step - loss: 0.0549 - accuracy: 0.9740 - val_loss: 1.3507 - val_accuracy: 0.7677\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 5s 25ms/step - loss: 0.0414 - accuracy: 0.9796 - val_loss: 1.2782 - val_accuracy: 0.7717\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6uf4_4y1aMYv",
        "outputId": "1b718d32-1f1d-43e6-8790-9482a5c03d2e"
      },
      "source": [
        "# Make predictions with our bidirectional model\n",
        "model_4_pred_probs = model_4.predict(val_sentences)\n",
        "model_4_pred_probs[:10], model_4_pred_probs.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[3.2934537e-03],\n",
              "        [7.6064408e-01],\n",
              "        [9.9996603e-01],\n",
              "        [2.9276967e-01],\n",
              "        [1.4514616e-05],\n",
              "        [9.9951208e-01],\n",
              "        [6.2776673e-01],\n",
              "        [9.9999046e-01],\n",
              "        [9.9994016e-01],\n",
              "        [5.7673228e-01]], dtype=float32), (762, 1))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8Q7Kh777aiNF",
        "outputId": "337e4c87-4e95-410f-d696-1b1f4a3819f5"
      },
      "source": [
        "# Convert pred probs to labels\n",
        "model_4_preds = tf.squeeze(tf.round(model_4_pred_probs))\n",
        "model_4_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g8wNtN5sarja",
        "outputId": "8d36d2aa-244e-43fc-cbb0-567b34b946fd"
      },
      "source": [
        "# Calculate model 4 results\n",
        "model_4_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_4_preds)\n",
        "model_4_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.16535433070865,\n",
              " 'f1': 0.7688960790251899,\n",
              " 'precision': 0.7747861668850706,\n",
              " 'recall': 0.7716535433070866}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4k4a1F7oatvK"
      },
      "source": [
        "## Model 5: Conv1D\n",
        "\n",
        "We've seen before how convolutional neural networks can be used for images but they can also be used for text.\n",
        "\n",
        "Previously we've used the layer Conv2D (which is great for images with (height, width)).\n",
        "\n",
        "But if we want to use convolutional layers for sequences (e.g. text) we need to use Conv1D: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D\n",
        "\n",
        "For more of a deep dive into what goes on behind the scenes in a CNN for text (or sequences) see the paper: https://arxiv.org/abs/1809.08037"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-ti4uulyc8s6",
        "outputId": "8f9fa0df-b2d4-4662-8600-2abcb76e7f96"
      },
      "source": [
        "# Let's do a little test to see what things look like through the eyes of a Conv1D layer\n",
        "embedding_test = embedding(text_vectorizer([\"this is a test sentence\"])) # turn target sentence into embedding\n",
        "conv_1d_layer = layers.Conv1D(filters=32,\n",
        "                              kernel_size=5, # setting this to 5 means it'll look at 5 words at a time, 3 would mean 3 words at a time\n",
        "                              activation=\"relu\")\n",
        "conv_1d_output = conv_1d_layer(embedding_test)\n",
        "max_pool = layers.GlobalMaxPool1D()\n",
        "max_pool_output = max_pool(conv_1d_output)\n",
        "print(f\"Embedding output shape: {embedding_test.shape}\")\n",
        "print(f\"Conv1D output shape: {conv_1d_output.shape}\")\n",
        "print(f\"Max pool output shape: {max_pool_output.shape}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Embedding output shape: (1, 15, 128)\n",
            "Conv1D output shape: (1, 11, 32)\n",
            "Max pool output shape: (1, 32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LzhUsJ5Te2qG",
        "outputId": "7d19e15c-65f4-47db-df93-bf95b583fa3c"
      },
      "source": [
        "# Let's see the outputs of each layer\n",
        "print(f\"Embedding output: {embedding_test}\")\n",
        "print(f\"Conv1D output: {conv_1d_output}\")\n",
        "print(f\"Max pool output: {max_pool_output}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Embedding output: [[[-0.00336894  0.02530373  0.03439075 ...  0.03439262 -0.01330827\n",
            "   -0.00163012]\n",
            "  [ 0.02095926 -0.03730217  0.04140642 ...  0.0258974  -0.01377566\n",
            "    0.00862277]\n",
            "  [ 0.01086491 -0.00704076 -0.032464   ... -0.01740124  0.05423632\n",
            "   -0.02601008]\n",
            "  ...\n",
            "  [-0.01084766 -0.01796961 -0.01566079 ...  0.00845763  0.01834772\n",
            "   -0.02773598]\n",
            "  [-0.01084766 -0.01796961 -0.01566079 ...  0.00845763  0.01834772\n",
            "   -0.02773598]\n",
            "  [-0.01084766 -0.01796961 -0.01566079 ...  0.00845763  0.01834772\n",
            "   -0.02773598]]]\n",
            "Conv1D output: [[[0.02085471 0.         0.0251439  0.01594924 0.03987276 0.\n",
            "   0.         0.         0.03106996 0.0693481  0.08296497 0.04628621\n",
            "   0.         0.02468291 0.         0.06159209 0.         0.\n",
            "   0.06167379 0.         0.         0.         0.00349523 0.\n",
            "   0.         0.         0.         0.         0.         0.\n",
            "   0.         0.        ]\n",
            "  [0.00054212 0.         0.05562776 0.         0.         0.\n",
            "   0.         0.07398236 0.         0.         0.         0.\n",
            "   0.00784798 0.13731796 0.         0.         0.         0.00929163\n",
            "   0.         0.         0.08877808 0.         0.         0.02935873\n",
            "   0.03264053 0.03866933 0.05365412 0.01732465 0.         0.06954296\n",
            "   0.00980187 0.0285482 ]\n",
            "  [0.0238361  0.02302316 0.03229465 0.         0.06289217 0.08523811\n",
            "   0.01136128 0.         0.05738889 0.         0.02491057 0.\n",
            "   0.         0.03656951 0.         0.04002353 0.         0.03880226\n",
            "   0.124845   0.         0.0087536  0.02037644 0.04507983 0.\n",
            "   0.         0.02188902 0.03331593 0.01122567 0.         0.02257476\n",
            "   0.00109781 0.05136009]\n",
            "  [0.05183294 0.         0.05846345 0.         0.04930804 0.\n",
            "   0.00796415 0.03070185 0.04773549 0.         0.         0.00065365\n",
            "   0.         0.03556914 0.         0.         0.         0.\n",
            "   0.         0.03211213 0.0578687  0.02963392 0.07651699 0.\n",
            "   0.         0.08474677 0.08265662 0.         0.04244438 0.\n",
            "   0.         0.00103243]\n",
            "  [0.05243194 0.         0.         0.         0.01205003 0.03402096\n",
            "   0.         0.00627384 0.09277191 0.         0.         0.02730833\n",
            "   0.         0.02347868 0.04263006 0.         0.         0.00583053\n",
            "   0.01346485 0.01151112 0.02323712 0.         0.04228698 0.\n",
            "   0.01047696 0.01118063 0.04947374 0.         0.03180653 0.05477148\n",
            "   0.         0.        ]\n",
            "  [0.04260373 0.00857761 0.         0.         0.00173682 0.05755334\n",
            "   0.019162   0.         0.06449309 0.         0.         0.\n",
            "   0.         0.02023829 0.02328368 0.00106196 0.         0.\n",
            "   0.00576173 0.00176063 0.00524058 0.         0.02950365 0.\n",
            "   0.00385522 0.01764445 0.02112489 0.         0.01462784 0.03994107\n",
            "   0.         0.        ]\n",
            "  [0.04260373 0.00857761 0.         0.         0.00173682 0.05755335\n",
            "   0.019162   0.         0.06449309 0.         0.         0.\n",
            "   0.         0.02023829 0.02328368 0.00106197 0.         0.\n",
            "   0.00576173 0.00176064 0.00524058 0.         0.02950365 0.\n",
            "   0.00385523 0.01764445 0.02112488 0.         0.01462784 0.03994108\n",
            "   0.         0.        ]\n",
            "  [0.04260372 0.00857762 0.         0.         0.00173682 0.05755336\n",
            "   0.01916201 0.         0.06449309 0.         0.         0.\n",
            "   0.         0.02023828 0.02328368 0.00106196 0.         0.\n",
            "   0.00576173 0.00176064 0.00524058 0.         0.02950365 0.\n",
            "   0.00385522 0.01764445 0.02112488 0.         0.01462783 0.03994108\n",
            "   0.         0.        ]\n",
            "  [0.04260372 0.00857762 0.         0.         0.00173682 0.05755335\n",
            "   0.019162   0.         0.06449309 0.         0.         0.\n",
            "   0.         0.02023829 0.02328369 0.00106197 0.         0.\n",
            "   0.00576173 0.00176063 0.00524058 0.         0.02950365 0.\n",
            "   0.00385522 0.01764444 0.02112488 0.         0.01462784 0.03994108\n",
            "   0.         0.        ]\n",
            "  [0.04260373 0.00857761 0.         0.         0.00173681 0.05755335\n",
            "   0.019162   0.         0.06449309 0.         0.         0.\n",
            "   0.         0.02023829 0.02328368 0.00106197 0.         0.\n",
            "   0.00576173 0.00176063 0.00524058 0.         0.02950365 0.\n",
            "   0.00385523 0.01764444 0.02112488 0.         0.01462784 0.03994108\n",
            "   0.         0.        ]\n",
            "  [0.04260372 0.00857761 0.         0.         0.00173682 0.05755335\n",
            "   0.01916201 0.         0.06449309 0.         0.         0.\n",
            "   0.         0.02023828 0.02328368 0.00106197 0.         0.\n",
            "   0.00576173 0.00176063 0.00524058 0.         0.02950365 0.\n",
            "   0.00385523 0.01764445 0.02112488 0.         0.01462784 0.03994108\n",
            "   0.         0.        ]]]\n",
            "Max pool output: [[0.05243194 0.02302316 0.05846345 0.01594924 0.06289217 0.08523811\n",
            "  0.01916201 0.07398236 0.09277191 0.0693481  0.08296497 0.04628621\n",
            "  0.00784798 0.13731796 0.04263006 0.06159209 0.         0.03880226\n",
            "  0.124845   0.03211213 0.08877808 0.02963392 0.07651699 0.02935873\n",
            "  0.03264053 0.08474677 0.08265662 0.01732465 0.04244438 0.06954296\n",
            "  0.00980187 0.05136009]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bh0b92eqfpCY",
        "outputId": "8bd81d40-eda2-4644-afce-61a6ce1f36e5"
      },
      "source": [
        "# Create 1-dimensional CNN to model sequences\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.Conv1D(filters=32, kernel_size=5, activation=\"relu\")(x)\n",
        "x = layers.GlobalMaxPool1D()(x)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\", name=\"output_layer\")(x)\n",
        "model_5 = tf.keras.Model(inputs, outputs, name=\"model_5_conv1d\")\n",
        "\n",
        "model_5.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_5_conv1d\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_5 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "conv1d_1 (Conv1D)            (None, 11, 32)            20512     \n",
            "_________________________________________________________________\n",
            "global_max_pooling1d_1 (Glob (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "output_layer (Dense)         (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 1,300,545\n",
            "Trainable params: 1,300,545\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k41K63I4iVz0"
      },
      "source": [
        "BATCH_SIZE = 32"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-sTeVF7thxRA",
        "outputId": "af18d3ef-2e6d-4523-bab1-1d425a53dde3"
      },
      "source": [
        "# Compile model\n",
        "model_5.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Fit the model\n",
        "history_model_5 = model_5.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              batch_size=BATCH_SIZE,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,\n",
        "                                                                     experiment_name=\"model_5_conv1d\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_5_conv1d/20210414-060027\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 6s 23ms/step - loss: 0.2130 - accuracy: 0.9489 - val_loss: 0.8363 - val_accuracy: 0.7795\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.0758 - accuracy: 0.9731 - val_loss: 0.9521 - val_accuracy: 0.7677\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.0668 - accuracy: 0.9738 - val_loss: 1.0429 - val_accuracy: 0.7690\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.0530 - accuracy: 0.9797 - val_loss: 1.1286 - val_accuracy: 0.7717\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.0446 - accuracy: 0.9820 - val_loss: 1.1877 - val_accuracy: 0.7651\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3sCcYEWNjYi5",
        "outputId": "b8622a66-43d2-46f5-f834-ef0c084c979a"
      },
      "source": [
        "# Make predictions with our 1D CNN\n",
        "model_5_pred_probs = model_5.predict(val_sentences)\n",
        "model_5_pred_probs[:5], model_5_pred_probs.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[4.6753362e-02],\n",
              "        [8.1121272e-01],\n",
              "        [9.9980360e-01],\n",
              "        [6.6667445e-02],\n",
              "        [2.9932426e-07]], dtype=float32), (762, 1))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XxJsKzVWjqhX",
        "outputId": "97570fd5-826e-44f0-ae33-efdfca315f9a"
      },
      "source": [
        "# Convert model 5 pred probs to labels\n",
        "model_5_preds = tf.squeeze(tf.round(model_5_pred_probs))\n",
        "model_5_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dOWYkGlZjqLK",
        "outputId": "bca5747f-5804-4ac3-a71f-0b9bc792c473"
      },
      "source": [
        "# Calculate results for model 5\n",
        "model_5_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_5_preds)\n",
        "model_5_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 76.50918635170603,\n",
              " 'f1': 0.7626195388168748,\n",
              " 'precision': 0.7672217903358366,\n",
              " 'recall': 0.7650918635170604}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kM3n8i-mj7hH"
      },
      "source": [
        "## Model 6: TensorFlow Hub Pretrained Sentence Encoder\n",
        "\n",
        "Now we've built a few of our own models, let's try and use transfer learning for NLP, specifically using TensorFlow Hub's Universal Sentence Encoder: https://tfhub.dev/google/universal-sentence-encoder/4\n",
        "\n",
        "See how the USE was created here: https://arxiv.org/abs/1803.11175\n",
        "\n",
        "📖 **Resource:** TensorFlow Hub is a great resource for many pretrained models but HuggingFace is also another incredible resource for many pretrained NLP models (using HuggingFace model is beyond the scope of this course but it is definitely something you should be familiar with in the NLP space): https://huggingface.co/models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "3K8Vuw-uG_ZJ",
        "outputId": "e553a614-d258-435b-94c7-5e9e8b9175f9"
      },
      "source": [
        "sample_sentence"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"There's a flood in my street!\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sdQhymwZD3Rp",
        "outputId": "2e1e387b-267b-4373-fb85-3cc6575cc038"
      },
      "source": [
        "import tensorflow_hub as hub\n",
        "embed = hub.load(\"https://tfhub.dev/google/universal-sentence-encoder/4\")\n",
        "embed_samples = embed([sample_sentence,\n",
        "                       \"When you can the universal sentence encoder on a sentence, it turns it into numbers.\"])\n",
        "print(embed_samples[0][:50])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[-0.01157024  0.0248591   0.0287805  -0.01271503  0.03971543  0.08827759\n",
            "  0.02680986  0.05589838 -0.01068732 -0.0059729   0.00639325 -0.01819523\n",
            "  0.00030817  0.09105892  0.05874645 -0.03180628  0.01512476 -0.05162928\n",
            "  0.00991369 -0.06865345 -0.04209306  0.02678981  0.03011009  0.00321069\n",
            " -0.00337969 -0.04787359  0.02266719 -0.00985924 -0.04063614 -0.01292095\n",
            " -0.04666384  0.056303   -0.03949255  0.00517685  0.02495828 -0.07014439\n",
            "  0.02871509  0.04947682 -0.00633971 -0.08960192  0.02807117 -0.00808362\n",
            " -0.01360601  0.05998649 -0.10361787 -0.05195372  0.00232955 -0.02332528\n",
            " -0.03758105  0.0332773 ], shape=(50,), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BFH2uFBcGzH8",
        "outputId": "fc4769e3-da05-4cea-ce78-2e4f8813b1ff"
      },
      "source": [
        "embed_samples[0].shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([512])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mZOcXMLiG-U2"
      },
      "source": [
        "# Create a Keras Layer using the USE pretrained layer from tensorflow hub\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[],\n",
        "                                        dtype=tf.string,\n",
        "                                        trainable=False,\n",
        "                                        name=\"USE\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DONRMBLXJcAa",
        "outputId": "8b810e73-8144-4149-f5a3-d6a2ae9c71dd"
      },
      "source": [
        "# Create model using the Sequential API\n",
        "model_6 = tf.keras.Sequential([\n",
        "  sentence_encoder_layer,\n",
        "  layers.Dense(64, activation=\"relu\"),\n",
        "  layers.Dense(1, activation=\"sigmoid\", name=\"output_layer\")                               \n",
        "], name=\"model_6_USE\")\n",
        "\n",
        "# Compile\n",
        "model_6.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "model_6.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "USE (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "output_layer (Dense)         (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7uALDPUuJ3Cx",
        "outputId": "a6dc8bd8-8576-464c-bc31-9a71b4d95350"
      },
      "source": [
        "# Train a classifier on top of USE pretrained embeddings\n",
        "model_6_history = model_6.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                     \"tf_hub_sentence_encoder\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/tf_hub_sentence_encoder/20210414-060134\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 10s 31ms/step - loss: 0.5727 - accuracy: 0.7393 - val_loss: 0.4550 - val_accuracy: 0.8058\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 16ms/step - loss: 0.4248 - accuracy: 0.8097 - val_loss: 0.4380 - val_accuracy: 0.8123\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 16ms/step - loss: 0.3925 - accuracy: 0.8250 - val_loss: 0.4340 - val_accuracy: 0.8097\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 15ms/step - loss: 0.3877 - accuracy: 0.8305 - val_loss: 0.4329 - val_accuracy: 0.8123\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 3s 15ms/step - loss: 0.3762 - accuracy: 0.8364 - val_loss: 0.4289 - val_accuracy: 0.8150\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uDrKgxzuKhtC",
        "outputId": "2698d86c-b348-4e1d-9a0a-c4ae13316269"
      },
      "source": [
        "# Make predictions with USE TF Hub Model\n",
        "model_6_pred_probs = model_6.predict(val_sentences)\n",
        "model_6_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.16757499],\n",
              "       [0.7972953 ],\n",
              "       [0.985545  ],\n",
              "       [0.19845757],\n",
              "       [0.75607044],\n",
              "       [0.7554092 ],\n",
              "       [0.9778758 ],\n",
              "       [0.977767  ],\n",
              "       [0.93762237],\n",
              "       [0.0849001 ]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W3J2sShbKwfs",
        "outputId": "2b0b3ad5-7ad2-4dfb-adb3-28a690451e3e"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_6_preds = tf.squeeze(tf.round(model_6_pred_probs))\n",
        "model_6_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ph8KI1oNK4HD",
        "outputId": "cf351a8d-934b-40e7-e967-0c0124160c3a"
      },
      "source": [
        "# Calculate model 6 performance metrics\n",
        "model_6_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_6_preds)\n",
        "model_6_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 81.49606299212599,\n",
              " 'f1': 0.8134357776936025,\n",
              " 'precision': 0.8172549323109193,\n",
              " 'recall': 0.8149606299212598}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gg41ex0sLDoF"
      },
      "source": [
        "## Model 7: TF Hub Pretrained USE but with 10% of training data\n",
        "\n",
        "Transfer learning really helps when you don't have a large dataset.\n",
        "\n",
        "To see how our model performs on a smaller dataset, let's replicate `model_6` except we'll train it on 10% of the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PZL8js9-M5oZ"
      },
      "source": [
        "# ## NOTE: Making data splits like below leads to data leakage (model_7 trained on 10% data, outperforms model_6 trained on 100% data)\n",
        "# ## DO NOT MAKE DATA SPLITS WHICH LEAK DATA FROM VALIDATION/TEST SETS INTO TRAINING SET \n",
        "\n",
        "# # Create subsets of 10% of the training data\n",
        "# train_10_percent = train_df_shuffled[[\"text\", \"target\"]].sample(frac=0.1, random_state=42)\n",
        "# # train_10_percent.head(), len(train_10_percent)\n",
        "# train_sentences_10_percent = train_10_percent[\"text\"].to_list()\n",
        "# train_labels_10_percent = train_10_percent[\"target\"].to_list()\n",
        "# len(train_sentences_10_percent), len(train_labels_10_percent)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "55bzxeJQUTC0"
      },
      "source": [
        "> 🔑 **Note:** Be *very* careful when creating training/val/test splits that you don't leak data across the datasets, otherwise your model evaluation metrics will be wrong. If something looks too good to be true (a model trained on 10% of data outperforming the same model trained on 100% of data) trust your gut and go back through to find where the error may lie."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CvoPRskfS8Pm"
      },
      "source": [
        "# Making a better dataset split (no data leakage)\n",
        "train_10_percent_split = int(0.1 * len(train_sentences))\n",
        "train_sentences_10_percent = train_sentences[:train_10_percent_split]\n",
        "train_labels_10_percent = train_labels[:train_10_percent_split]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oglhBYknTKAL",
        "outputId": "79a1930a-9003-411f-c74b-7c34115be4ba"
      },
      "source": [
        "# Check the number of each label in the updated training data subset\n",
        "pd.Series(np.array(train_labels_10_percent)).value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    406\n",
              "1    279\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gGNtG6izNM80",
        "outputId": "89758ae9-297b-47fb-c2e5-6117fda39897"
      },
      "source": [
        "# Check the number of targets in our subset of data\n",
        "train_df_shuffled[\"target\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjxT2OEjOzm2"
      },
      "source": [
        "To recreate a model the same as a previous model you've created you can use the `tf.keras.models.clone_model()` method, see more here: https://www.tensorflow.org/api_docs/python/tf/keras/models/clone_model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GTAkNUBvNjBx",
        "outputId": "92edfc47-e72a-45a6-e202-48b7d8ebe2a2"
      },
      "source": [
        "# Let's build a model the same as model_6\n",
        "# model_7 = tf.keras.models.clone_model(model_6)\n",
        "model_7 = tf.keras.Sequential([\n",
        "  sentence_encoder_layer,\n",
        "  layers.Dense(64, activation=\"relu\"),\n",
        "  layers.Dense(1, activation=\"sigmoid\", name=\"output_layer\")                               \n",
        "], name=\"model_7_USE\")\n",
        "\n",
        "# Compile model\n",
        "model_7.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get a summary (will be same as model_6)\n",
        "model_7.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_7_USE\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "USE (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "output_layer (Dense)         (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Egc7fLk9N-JP",
        "outputId": "bdb257d3-275c-45aa-f4cb-48c3cbc4a694"
      },
      "source": [
        "# Fit the model to the 10% training data subsets\n",
        "model_7_history = model_7.fit(train_sentences_10_percent,\n",
        "                              train_labels_10_percent,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                     \"tf_hub_sentence_encoder_10_percent_correct_split\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: model_logs/tf_hub_sentence_encoder_10_percent_correct_split/20210414-060159\n",
            "Epoch 1/5\n",
            "22/22 [==============================] - 3s 101ms/step - loss: 0.6793 - accuracy: 0.6465 - val_loss: 0.6444 - val_accuracy: 0.7480\n",
            "Epoch 2/5\n",
            "22/22 [==============================] - 1s 29ms/step - loss: 0.6032 - accuracy: 0.8443 - val_loss: 0.5847 - val_accuracy: 0.7730\n",
            "Epoch 3/5\n",
            "22/22 [==============================] - 1s 29ms/step - loss: 0.5272 - accuracy: 0.8262 - val_loss: 0.5307 - val_accuracy: 0.7848\n",
            "Epoch 4/5\n",
            "22/22 [==============================] - 1s 28ms/step - loss: 0.4612 - accuracy: 0.8385 - val_loss: 0.5019 - val_accuracy: 0.7730\n",
            "Epoch 5/5\n",
            "22/22 [==============================] - 1s 29ms/step - loss: 0.4213 - accuracy: 0.8317 - val_loss: 0.4878 - val_accuracy: 0.7769\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZzQaiu_kQH22",
        "outputId": "37c2df76-d40c-4d50-d909-cf93622c4133"
      },
      "source": [
        "# Make predictions with the model trained on 10% of the data\n",
        "model_7_pred_probs = model_7.predict(val_sentences)\n",
        "model_7_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.2083207 ],\n",
              "       [0.6036797 ],\n",
              "       [0.92795014],\n",
              "       [0.37632278],\n",
              "       [0.5633504 ],\n",
              "       [0.6833203 ],\n",
              "       [0.8904111 ],\n",
              "       [0.80405366],\n",
              "       [0.85651606],\n",
              "       [0.15267055]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ma3xidC1QTL1",
        "outputId": "66a1f341-49a9-4961-8862-c581baa00a19"
      },
      "source": [
        "# Turn pred probs into labels\n",
        "model_7_preds = tf.squeeze(tf.round(model_7_pred_probs))\n",
        "model_7_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 91
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JG4rtLi2QcTm",
        "outputId": "953ca2af-ca9b-42e4-8375-95a8f3ee0ca7"
      },
      "source": [
        "# Evalaute model 7 predictions\n",
        "model_7_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_7_preds)\n",
        "model_7_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.69028871391076,\n",
              " 'f1': 0.7758759837011532,\n",
              " 'precision': 0.7769803060533722,\n",
              " 'recall': 0.7769028871391076}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26PsF3DuRQrt"
      },
      "source": [
        "## Comparing the peformance of each of our models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "id": "MgI0SBZMVSnQ",
        "outputId": "e8442e34-6e5e-4b4d-f249-94634e25bf81"
      },
      "source": [
        "# Combine model results into a DataFrame\n",
        "all_model_results = pd.DataFrame({\"0_baseline\": baseline_results,\n",
        "                                  \"1_simple_dense\": model_1_results,\n",
        "                                  \"2_lstm\": model_2_results,\n",
        "                                  \"3_gru\": model_3_results,\n",
        "                                  \"4_bidirectional\": model_4_results,\n",
        "                                  \"5_conv1d\": model_5_results,\n",
        "                                  \"6_tf_hub_use_encoder\": model_6_results,\n",
        "                                  \"7_tf_hub_use_encoder_10_percent\": model_7_results})\n",
        "all_model_results = all_model_results.transpose()\n",
        "all_model_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0_baseline</th>\n",
              "      <td>79.265092</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1_simple_dense</th>\n",
              "      <td>79.133858</td>\n",
              "      <td>0.797601</td>\n",
              "      <td>0.791339</td>\n",
              "      <td>0.788036</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2_lstm</th>\n",
              "      <td>78.215223</td>\n",
              "      <td>0.784069</td>\n",
              "      <td>0.782152</td>\n",
              "      <td>0.780177</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3_gru</th>\n",
              "      <td>76.246719</td>\n",
              "      <td>0.770150</td>\n",
              "      <td>0.762467</td>\n",
              "      <td>0.757605</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4_bidirectional</th>\n",
              "      <td>77.165354</td>\n",
              "      <td>0.774786</td>\n",
              "      <td>0.771654</td>\n",
              "      <td>0.768896</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5_conv1d</th>\n",
              "      <td>76.509186</td>\n",
              "      <td>0.767222</td>\n",
              "      <td>0.765092</td>\n",
              "      <td>0.762620</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6_tf_hub_use_encoder</th>\n",
              "      <td>81.496063</td>\n",
              "      <td>0.817255</td>\n",
              "      <td>0.814961</td>\n",
              "      <td>0.813436</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7_tf_hub_use_encoder_10_percent</th>\n",
              "      <td>77.690289</td>\n",
              "      <td>0.776980</td>\n",
              "      <td>0.776903</td>\n",
              "      <td>0.775876</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  accuracy  precision    recall        f1\n",
              "0_baseline                       79.265092   0.811139  0.792651  0.786219\n",
              "1_simple_dense                   79.133858   0.797601  0.791339  0.788036\n",
              "2_lstm                           78.215223   0.784069  0.782152  0.780177\n",
              "3_gru                            76.246719   0.770150  0.762467  0.757605\n",
              "4_bidirectional                  77.165354   0.774786  0.771654  0.768896\n",
              "5_conv1d                         76.509186   0.767222  0.765092  0.762620\n",
              "6_tf_hub_use_encoder             81.496063   0.817255  0.814961  0.813436\n",
              "7_tf_hub_use_encoder_10_percent  77.690289   0.776980  0.776903  0.775876"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ecgLCp4IVkOX"
      },
      "source": [
        "# Reduce the accuracy to the same scale as other metrics\n",
        "all_model_results[\"accuracy\"] = all_model_results[\"accuracy\"]/100\n",
        "# all_model_results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        },
        "id": "LprTMBfQWckZ",
        "outputId": "8c8636f4-7731-4695-d966-973deb9a1872"
      },
      "source": [
        "# Plot and compare all of the model results\n",
        "all_model_results.plot(kind=\"bar\", figsize=(10, 7)).legend(bbox_to_anchor=(1.0, 1.0));"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAI9CAYAAAAZ0eGSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde5yWdZ3/8fd7OIjIQcWRUEBQERgVRZHMQ5aH0lXR0k1M09qKtQ017US1alFmmmnRuruYmeZhzcwUj2il8Cs1wQPKQRSREBQdFUFFgoHP74/rGrkZBmbQYa7vcL2ej8c8uK/D3POZ+8HMvO/v0REhAAAAICVVRRcAAAAANERIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOe2L+sLbbbdd9OvXr6gvDwAA0GyPPfbYaxFRXXQdZVJYSO3Xr5+mTp1a1JcHAABoNtv/KLqGsqG7HwAAAMkhpAIAACA5hFQAAAAkp7AxqQAAAG3ZY489tn379u2vkrSHaPjbWKslTa+rq/vSvvvu+2pjNxBSAQAA3of27dtf9aEPfWhwdXX14qqqqii6nrZk9erVrq2trVm0aNFVkkY0dg+pHwAA4P3Zo7q6eikBdeNVVVVFdXX1EmWt0I3f04r1AAAAbE6qCKjvX/7arTeLElIBAACQHMakAgAAtIB+Y+7atyWfb95Pjn6sJZ+vraElFQAAABu0cuXKVv+ahFQAAIA27PDDD99l9913H7zrrrvufumll24nSbfccku3mpqawQMHDqz5yEc+spskLVmypOrEE0/st9tuu9XstttuNddcc83WktS5c+eh9c/1m9/8ZpsTTjihnySdcMIJ/T772c/2HTJkyKCvfOUrvR944IHOe++996DBgwfXDB06dNC0adO2kKS6ujqNGjWq94ABA3bfbbfdai688MLtJ0yY0PXwww/fpf55//jHP3Y74ogjdtFGoLsfAACgDbvhhhvm9ezZc9Xbb7/toUOH1px00klvjh49ut+DDz74zKBBg1a88sor7SRpzJgxvbp167bq2WefnSlJtbW17Zp67pdffrnj448//kz79u31xhtvVE2ZMuWZDh066Lbbbuv6rW99q/fEiROf/9nPflY9f/78jjNnzpzRoUMHvfLKK+2qq6tXnX322X1feuml9jvssEPd1Vdf3eMLX/jCaxvzfRFSAQAA2rCLL76451133bW1JC1atKjDuHHjqocPH/7WoEGDVkhSz549V0nS5MmTu910001z6z+vurp6VVPP/elPf3px+/ZZXHzjjTfanXTSSf3nzZvXyXasXLnSkvSXv/yl2xlnnFHboUMHVX69z3zmM6//6le/2varX/3q648//niXW2+99YWN+b4IqQAAAG3UnXfe2XXSpEldp06d+kzXrl1XDx8+fODQoUOXzZ49u1Nzn8P2e4/fffddV17r0qXL6vrH3/72t3c85JBD3rr//vufnz17dsdDDz104Iae9ytf+crrRx999K6dOnWKY489dnF9iG0uxqQCAAC0UW+++Wa77t27r+ratevqJ554otO0adO2Wr58edWjjz7a9ZlnnukoSfXd/YcccsjSyy+/fPv6z63v7u/Ro8fKxx9/vNOqVat0++23b7O+r7V06dJ2vXv3XiFJ48eP367+/GGHHbZ0/Pjx29VPrqr/ev369VvZs2fPlT/72c96jRo1aqO6+iVaUgEAAFpEEUtGnXDCCUuuvPLK6p133nn3nXfeeflee+31zvbbb183bty4eZ/61Kd2Xb16tXr06LHyoYceeu6iiy56+Qtf+ELfAQMG7F5VVRXf/e53Xzr99NPf/MEPfrDwuOOO23Xbbbet22uvvZa98847jTZifvvb3170pS99qf/FF1+8wxFHHPFm/flzzjmn9tlnn91i0KBBu7dv3z5OP/302u9+97u1kjRy5MjXr7jiivb77LPP8o393hxRzEYJw4YNi6lTpxbytQEAaFXf797E9SWtUwfeN9uPRcSwynPTpk2bt9dee210C2GZnHbaaX2HDh267Jxzzmn0dZo2bdp2e+21V7/GrtGSCgDAB9BvzF1N3jOvidGBe167Z5PP8fTpTze3JCAJu+++++Att9xy9fjx4198P59PSAUAoA2YNWhwk/cMfmZWK1QCNM+MGTM+0H/IzT+kNtXFItHNAgAAkBhm9wMAACA5zQqpto+0Pdv2HNtjGrne1/YDtp+w/ZTtf2n5UgEAAFAWTYZU2+0kXSHpKEk1kk62XdPgtv+UdHNEDJU0UtJ/t3ShAAAAKI/mjEkdLmlORMyVJNs3STpO0syKe0JSt/xxd0kvtWSRAAAAyft+931b9vmWtPq6q5I0efLkzldffXWPa665ptFZ+fPmzetwxhln9Ln33nvnNna9pTQnpO4oqbLIBZI+3OCe70u6z/aZkraSdHhjT2R7lKRRktS3b9+NrRUAAAAbqa6uTu3bN3+u/Ec/+tFlH/3oR5et73q/fv1WbuqAKrXcxKmTJV0TEb0l/Yuk62yv89wRcWVEDIuIYdXV1S30pfG+fL970x8AACBps2fP7ti/f//dR4wY0X/nnXfe/cgjj9z5rbfeqtpxxx33/MpXvrJjTU3N4KuvvnqbW2+9tdvee+89qKamZvBRRx2185IlS6okadKkSZ2HDh06aODAgTV77rnn4MWLF1fdeeedXT/+8Y/vKkl33XVXl0GDBtUMGjSoZvDgwTWLFy+umj17dscBAwbsLknLli3ziSee2G+33XarGTx4cM0dd9zRVZLGjRvX4xOf+MQuBx988ICddtppjzPOOKP3xn5vzQmpCyX1qTjunZ+r9EVJN0tSRDwsqZOk7QQAAIBNat68eZ1Gjx796ty5c2d07dp19U9/+tNqSerRo0fdzJkzZx177LFv/fjHP+41efLkZ2fOnDlrn332WfbDH/6w5/Lly33KKafs8vOf/3z+7NmzZ06aNGl2ly5dVlc+989+9rMPjRs37h/PPPPMzEceeeSZhtcvvvji7W3r2WefnXnjjTfOHTVqVL9ly5ZZkmbOnNn5tttumztr1qwZEyZM2GbOnDkdNub7ak7b7xRJA2z3VxZOR0r6bIN75ks6TNI1tgcrC6m1G1PI+9XUTh9N7fIhsdMHAABouz70oQ+t+MQnPvGOJH3uc597fdy4cdtL0mmnnbZYkh588MGtnn/++U7Dhw8fJEkrV670vvvu+/ZTTz3Vafvtt195yCGHLJOkbbfddnXD595///3f/sY3vtHnM5/5zBsnn3zy4l122WWtex566KEuZ5555quSNHTo0OU77LDDiqeffrqTJB100EFLe/TosUqSdt111+XPP//8FrvuuuvK5n5fTYbUiKizPVrSREntJF0dETNsj5U0NSImSPq6pF/ZPkfZJKrPR0Q0twi0rJbYok9qOrwT3AEAKJ7tRo+7du26WpIiQgcddNDSO+6444XK+x599NEtm3ruH//4x4uOP/74Jbfffnv3gw8+eNBdd931XOfOndcJs43p2LHje1mwXbt2sXLlSm/o/oaaNSY1Iu6OiN0iYpeIuDA/d34eUBURMyPiwIjYKyL2joj7NqYItE2zBg1u8gMAAGxaL7/8csc//elPW0nSDTfcsO0BBxzwduX1j33sY+9MnTq1y/Tp07eQpKVLl1Y99dRTWwwZMmT5q6++2mHSpEmdJWnx4sVVK1eu3dA5Y8aMLYYPH/7uhRdeuGjIkCHvTJ8+fa1mrgMPPPDt66+/fltJeuqpp7Z4+eWXOw4ZMmR5S3xfm/+2qC2kqcDFfskAAJRcQUtG9evXb/kvf/nL7UeNGtV5wIABy7/xjW/UXnXVVdvXX99hhx3qxo8fP2/kyJE7r1ixwpJ0wQUXLBwyZMg/b7jhhufPOuusvsuXL6/q1KnT6smTJz9b+dyXXHLJ9g899FA32zFw4MB3TzzxxCXz589/b2zpt771rVdPO+20nXbbbbeadu3aafz48fO23HLLFulNJ6QClZpa1eD7S1qnDgAAmql9+/a6/fbb1+rKX7hw4Vpj8kaMGPHWiBEj1mlRO+SQQ5ZNmzbtmcpzxxxzzFvHHHPMW5J07bXXrrNW6sCBA1c899xzMySpc+fOccstt8xreM9ZZ531uqTX648feOCBORv3XRFSUSItMVaXSXYAALQOQirQwpozFpfhIQCAllDZqrm5IaQCAIAW1/QSkQ1Xs1zXnv2b3p2S3qvNFyEVAN6P5uzKxhhmYJOj92rz1VLbogIAAAAthpZUABtW0hZDdrMDgGIRUgGgQKzBDGw+9rx2z31b8vmePv3pQtZdHTduXI+pU6du9dvf/nb+ueeeu0OXLl1WjR079pXWroOQCgBoOSVteQdSsHr1akWE2rVrV3QpLYKQCgBolpZYa1hqehgEQyCA5ps9e3bHT37yk7sNHTr07aeffnqr44477o2JEyduvWLFCh999NFvXn755S9J0n/913/1GDduXE/bGjx48Lu33XbbCzfeeGP3n/zkJ71WrlxZtc0229T97ne/m9unT5+6or+neoRUoOQYe4nUMFsb2Djz58/f4te//vULS5YseeP3v//9Nk899dSsiNDhhx++6z333NOlurq67tJLL+318MMPP9OrV6+6V155pZ0kHXHEEW+PHDnymaqqKl122WXbjR079kO/+tWvFhT9/dQjpAIAALRhvXr1WnHYYYe9M2rUqN6TJ0/uVlNTUyNJy5Ytq3rmmWc6Pf7441XHHnvs4l69etVJUs+ePVdJ0gsvvNDx+OOP711bW9thxYoVVX369Plnkd9HQ4RUAK2C1jEA2DQ6d+68WpIiQl/72tde/uY3v/la5fULL7xw+8Y+b/To0X3PPvvsRaeccsqSO++8s+vYsWN3aI16m4t1UgEAADYDRx111NLrrrtuuyVLllRJ0gsvvNBh4cKF7T/5yU8uveOOO7ZZtGhRO0mq7+5/66232vXt23elJF1zzTU9iqu8cbSkAgAAtICiloyq9+lPf3rpjBkzOu23336DpKyF9YYbbnhh2LBhy7/+9a+/fPDBBw+qqqqKPfbYY9kf/vCHed/73vdeOvnkk3fp3r173UEHHfTW/Pnztyiy/oYIqQAAAG3UwIEDVzz33HMz6o/PO++8V88777xXG9535plnvn7mmWe+Xnnu1FNPffPUU099s+G9Z5111uuSXpekyy677KVNUHaz0N0PAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAyWEJKgAAgBYwa9DgfVvy+QY/M6vJdVd/9KMfbX/11VdXDxgwYPkrr7zSYebMmZ3HjBmzcOzYsa+0ZC1FIKQCAAC0Ub/+9a+r//SnPz3bqVOnmDNnTsdbbrllm6Jrail09wMAALRBn/3sZ/suWLBgi6OOOmrAVVddte0hhxyyrEOHDlF0XS2FllQAAIA26MYbb5w/adKk7pMmTXq2V69edUXX09JoSQUAAEByCKkAAABIDiEVAAAAyWFMKgAAQAtozpJRm8r8+fPb77fffjXvvPNOO9sxfvz4nrNmzZq+7bbbri6qpg+KkAoAANBGLVy48On6x6+88spTRdbS0ujuBwAAQHIIqQAAAEgOIRUAAOD9Wb169WoXXURblb926x0z26yQavtI27Ntz7E9ppHrl9t+Mv941vabH6BmAACAtmB6bW1td4Lqxlu9erVra2u7S5q+vnuanDhlu52kKyQdIWmBpCm2J0TEzPp7IuKcivvPlDT0gxQOAACQurq6ui8tWrToqkWLFu0heqc31mpJ0+vq6r60vhuaM7t/uKQ5ETFXkmzfJOk4STPXc//Jki7YyEIBAADalH333fdVSSOKrmNz1ZzUv6OkFyuOF+Tn1mF7J0n9Jf1lPddH2Z5qe2ptbe3G1goAAICSaOmm6ZGSbomIVY1djIgrI2JYRAyrrq5u4S8NAACAzUVzQupCSX0qjnvn5xozUtL/fdCiAAAAUG7NCalTJA2w3d92R2VBdELDm2wPkrSNpIdbtkQAAACUTZMhNSLqJI2WNFHSLEk3R8QM22NtVw4WHinppoiITVMqAAAAyqI5s/sVEXdLurvBufMbHH+/5coCAABAmbGmFwAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJaVZItX2k7dm259ges557PmN7pu0Ztm9s2TIBAABQJu2busF2O0lXSDpC0gJJU2xPiIiZFfcMkPQdSQdGxGLb22+qggEAALD5a05L6nBJcyJibkSskHSTpOMa3PNlSVdExGJJiohXW7ZMAAAAlElzQuqOkl6sOF6Qn6u0m6TdbP/N9iO2j2zsiWyPsj3V9tTa2tr3VzEAAAA2ey01caq9pAGSPibpZEm/sr11w5si4sqIGBYRw6qrq1voSwMAAGBz05yQulBSn4rj3vm5SgskTYiIlRHxgqRnlYVWAAAAYKM1J6ROkTTAdn/bHSWNlDShwT23KWtFle3tlHX/z23BOgEAAFAiTYbUiKiTNFrSREmzJN0cETNsj7U9Ir9toqTXbc+U9ICkb0bE65uqaAAAAGzemlyCSpIi4m5Jdzc4d37F45B0bv4BAAAAfCDsOAUAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACS06yQavtI27Ntz7E9ppHrn7dda/vJ/ONLLV8qAAAAyqJ9UzfYbifpCklHSFogaYrtCRExs8Gtv4uI0ZugRgAAAJRMc1pSh0uaExFzI2KFpJskHbdpywIAAECZNSek7ijpxYrjBfm5hk6w/ZTtW2z3aeyJbI+yPdX21Nra2vdRLgAAAMqgpSZO3SGpX0QMkXS/pGsbuykiroyIYRExrLq6uoW+NAAAADY3zQmpCyVVtoz2zs+9JyJej4h/5odXSdq3ZcoDAABAGTUnpE6RNMB2f9sdJY2UNKHyBtu9Kg5HSJrVciUCAACgbJqc3R8RdbZHS5ooqZ2kqyNihu2xkqZGxARJZ9keIalO0huSPr8JawYAAMBmrsmQKkkRcbekuxucO7/i8XckfadlSwMAAEBZseMUAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASE6zQqrtI23Ptj3H9pgN3HeC7bA9rOVKBAAAQNk0GVJtt5N0haSjJNVIOtl2TSP3dZV0tqS/t3SRAAAAKJfmtKQOlzQnIuZGxApJN0k6rpH7fijpYknLW7A+AAAAlFBzQuqOkl6sOF6Qn3uP7X0k9YmIuzb0RLZH2Z5qe2ptbe1GFwsAAIBy+MATp2xXSbpM0tebujciroyIYRExrLq6+oN+aQAAAGymmhNSF0rqU3HcOz9Xr6ukPSQ9aHuepP0lTWDyFAAAAN6v5oTUKZIG2O5vu6OkkZIm1F+MiCURsV1E9IuIfpIekTQiIqZukooBAACw2WsypEZEnaTRkiZKmiXp5oiYYXus7RGbukAAAACUT/vm3BQRd0u6u8G589dz78c+eFkAAAAoM3acAgAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5zQqpto+0Pdv2HNtjGrl+hu2nbT9p+6+2a1q+VAAAAJRFkyHVdjtJV0g6SlKNpJMbCaE3RsSeEbG3pEskXdbilQIAAKA0mtOSOlzSnIiYGxErJN0k6bjKGyJiacXhVpKi5UoEAABA2bRvxj07Snqx4niBpA83vMn2VyWdK6mjpEMbeyLboySNkqS+fftubK0AAAAoiRabOBURV0TELpK+Lek/13PPlRExLCKGVVdXt9SXBgAAwGamOSF1oaQ+Fce983Prc5Ok4z9IUQAAACi35oTUKZIG2O5vu6OkkZImVN5ge0DF4dGSnmu5EgEAAFA2TY5JjYg626MlTZTUTtLVETHD9lhJUyNigqTRtg+XtFLSYkmnb8qiAQAAsHlrzsQpRcTdku5ucO78isdnt3BdAAAAKDF2nAIAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJaVZItX2k7dm259ge08j1c23PtP2U7T/b3qnlSwUAAEBZNBlSbbeTdIWkoyTVSDrZdk2D256QNCwihki6RdIlLV0oAAAAyqM5LanDJc2JiLkRsULSTZKOq7whIh6IiGX54SOSerdsmQAAACiT5oTUHSW9WHG8ID+3Pl+UdE9jF2yPsj3V9tTa2trmVwkAAIBSadGJU7ZPlTRM0k8bux4RV0bEsIgYVl1d3ZJfGgAAAJuR9s24Z6GkPhXHvfNza7F9uKTvSTokIv7ZMuUBAACgjJrTkjpF0gDb/W13lDRS0oTKG2wPlTRe0oiIeLXlywQAAECZNBlSI6JO0mhJEyXNknRzRMywPdb2iPy2n0rqIun3tp+0PWE9TwcAAAA0qTnd/YqIuyXd3eDc+RWPD2/hugAAAFBi7DgFAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAktOskGr7SNuzbc+xPaaR6x+1/bjtOtsntnyZAAAAKJMmQ6rtdpKukHSUpBpJJ9uuaXDbfEmfl3RjSxcIAACA8mnfjHuGS5oTEXMlyfZNko6TNLP+hoiYl19bvQlqBAAAQMk0p7t/R0kvVhwvyM9tNNujbE+1PbW2tvb9PAUAAABKoFUnTkXElRExLCKGVVdXt+aXBgAAQBvSnJC6UFKfiuPe+TkAAABgk2hOSJ0iaYDt/rY7ShopacKmLQsAAABl1mRIjYg6SaMlTZQ0S9LNETHD9ljbIyTJ9n62F0j6V0njbc/YlEUDAABg89ac2f2KiLsl3d3g3PkVj6coGwYAAAAAfGDsOAUAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcpoVUm0faXu27Tm2xzRyfQvbv8uv/912v5YuFAAAAOXRZEi13U7SFZKOklQj6WTbNQ1u+6KkxRGxq6TLJV3c0oUCAACgPJrTkjpc0pyImBsRKyTdJOm4BvccJ+na/PEtkg6z7ZYrEwAAAGXiiNjwDfaJko6MiC/lx5+T9OGIGF1xz/T8ngX58fP5Pa81eK5RkkblhwMlzW6pb+QD2k7Sa03eVT68LuviNWkcr0vjeF0ax+uyLl6TxqX0uuwUEdVFF1Em7Vvzi0XElZKubM2v2Ry2p0bEsKLrSA2vy7p4TRrH69I4XpfG8bqsi9ekcbwu5dac7v6FkvpUHPfOzzV6j+32krpLer0lCgQAAED5NCekTpE0wHZ/2x0ljZQ0ocE9EySdnj8+UdJfoqKRXHwAACAASURBVKlxBAAAAMB6NNndHxF1tkdLmiipnaSrI2KG7bGSpkbEBEm/lnSd7TmS3lAWZNuS5IYgJILXZV28Jo3jdWkcr0vjeF3WxWvSOF6XEmty4hQAAADQ2thxCgAAAMkhpAIAACA5hFQAAAAkh5AKAEABbFfZPqDoOoBUlXbilO3Okr4uqW9EfNn2AEkDI+LOgktLgu3OEbGs6DpSYnsbZesBv7cqRkQ8XlxFxbL90cbOR8Tk1q4lBbY/vaHrEXFra9WCtsP2ExExtOg6UmL7uoj4XFPnsPlr1R2nEvMbSY9J+kh+vFDS7yWVOqTm7+qvktRFUl/be0n694j4j2IrK5btH0r6vKTnJdW/swtJhxZVUwK+WfG4k6Thyn6myvqaHLuBayGplCHV9lta8zOzjojo1orlpOjPtk+QdCvri79n98oD2+0k7VtQLShQmVtSp0bEsMp3sbanRcReRddWJNt/V7Yhw4SK12V6ROxRbGXFsj1b0p4RsaLoWlJlu4+kn0fECUXXgvTkb/RelnSdJEs6RVKviDi/0MIKlof4rSStkvSustcmyhjebX9H0nclbSmpvifPklZIujIivlNUbShGmVtSV9jeUvk7fNu7SPpnsSWlISJetF15alVRtSRkuqStJb1adCEJWyBpcNFFpMD20cpagzrVn4uIscVVlIQRDRoB/sf2NEmlDqkR0bXoGlIRERdJusj2RQRSSOUOqRdIuldSH9s3SDpQWXdu2b2Yd/mH7Q6SzpY0q+CaUnCRpCdsT1fFm5mIGFFcScWy/Uut6catkrS3pNKO0a1n+38ldZb0cWVDZ06U9GihRaXhHdunSLpJ2f+bkyW9U2xJxXPWInCKpP4R8cO8R6JXRJT2/0xEfMf2jpJ20tpzAEo53r3MStvdL0m2e0jaX1l3wiMR8VrBJRXO9naSfiHpcGWvy32Szo6I1wstrGC2Z0gaL+lpSavrz0fEpMKKKpjt0ysO6yTNi4i/FVVPKmw/FRFDKv7tIumeiDi46NqKZLufst8tByoLqX+T9LWImFdcVcWz/T/KfqccGhGD8wma90XEfgWXVhjbP1G2vfpMrenJizI3CpRVmVtSpawrbrGy16HGdunfqeVB/ZSi60jQsogYV3QRqcgnMnwiIvi/sq5383+X2d5B0uuSehVYTxLyMHpc0XUk6MMRsY/tJyQpIhbb7lh0UQX7lLLVdhiCV3KlDam2L5Z0kqQZWtMyFpJKHVJtXyLpR8r+0N4raYikcyLi+kILK97/s32RpAlau7u/lN3bEbHK9k62OzKZbB132t5a0k+VDX8IZd3+aMD2+YzV1cr8TV/9/IhqVfTWlNRcSR3EPJHSK213fz5bewjv1NZm+8mI2Nv2pyQdI+lcSZNZ9cAPNHI6IqKsyy3J9m+VTZSaoIqxhRFxWWFFJcb2FpI6RcSSomtJke35EdG36DqKlI/TPUnSPpKuVTaG+T8j4veFFlYg23+QtJekP2vtRoGzCisKhShtS6p4p7Y+9f8njpb0+4hY0mCmf1l9MSLmVp6wvXNRxSTi+fyjShIzlCvkkw/7Kf95yocS/bbQogpie+n6LilbaqjUIuIG249JOkzZa3J8RJR9suqE/AMlV+aWVN6pNSIfsH68su7+4cqWXbozIj5caGEFs/14ROzT4NxjEcEC01iL7esk7SLpSa096aOUv1tsz5e0X0S80si1FyOiTwFlFc72thu6HhFvtFYtKcqXiOwbEbOLrgXFKXNLKu/UGhERY/JxqUvycYfvqMSTHWwPUrbeZfcG2152U8UamGVk+w6tu5PQEklTJY2PiOWtX1UShkmqYfeg9/xW2VJC64RUSTe2ci0peUzZz48l9VU2idfKGgbmS+pfXGnFsn2spEsldZTU3/beksYyu798StuSivVr2FUpqcxdlccpa1keobXf1Lwl6aaIeKiQwhJg+xeSqiX9X37qJElLlf3h7VbWfbZt/17SWRHxctG1IH22fyXpjxFxd358lLIu/38vtrLi5MMfDpX0IDsfllvpWlJt3xwRn7H9tBrZTzoihhRQVjLW11WprDWkdCLidkm32/5IRDxcdD2JOaDBWo532J4SEfvl68qW1XaSZtp+VGz88J685f3/JN0eEaVfxL/C/hHx5fqDiLgn780qs5WNzIco+4oHpVS6kKpsByUpm7mOddFV2bhP5cGLpbnW6GK7b0TMlyTbfSV1ya+VeVmq7xddQKIuVdbafpHtKcp2nrqzxMNC6r1k+z8l1f8uOUXSSwXWk4IZtj8rqZ3tAZLOklTaXqsyo7sfa6GrsnEszbUu2/8i6X+VzfC3sjF0/yHpQUlfjoifF1ddsWz3lFTfyvxoRLxaZD0pydcEPVTSlyUdGRHdCi6pUPkEqgskfTQ/NVnSD8o8ccp2Z0nfk/SJ/NREST/iDU35lC6k2n5La7r56/sS6gevB78w/YCyPdjpqqxge0ZE7G77Kkm3RMS9tqeVOaRK760DOig/nF35R8T2ERFxfzGVFcf2Z5Qt5P+gst8rB0v6ZkTcUmRdKchnbB+rNeuC3hkRZxZbVRpsd1X2N+jtomsBUlG6kIoNs31IY+fLvEe9xNJc70djy3aVge1pko6obz3NdxD6E29ofLOyn517Jf1O0qSIKP04Q9t7KhvzX78k1WuSTo+I6cVVVSzb90v614h4Mz/eRtlE1U8WWxlaWxnHpL7H9kGSBkTEb2xvJ6lrRLxQdF1FiohJtndS9rr8Ke92aVd0XUVjaa73pay7QFQ16N5/XdmGB2X3a0knR8SqJu8sl/GSzo2IByTJ9sckXSnpgCKLKth29QFVkiJise3tiywIxShtSLV9gbJJQgMl/UbZemzXSzqwyLqKZvvLkkYpe1e/i6QdlY07PKzIuorSYG3U+nOVh7e2XjVtTlm7ae61PVFrL811d4H1JCEiJto+wHY/sbxdpa3qA6okRcSDtrcqsqAErG4wKXMnlff3SamVNqRK+pSkoZIel6SIeCkfE1R2X1XWJfd3SYqI50r+DvbYDVwLEVLRQER80/YJWvOG98qI+GORNaWA5e3Wa67t8yRdlx+fqmzb7jL7rqS/2p6kNeO6RxVbEopQ5pC6IiLCdkgS71zf88+IWFHfWmi7vUr8DjYivtCc+2yfHhHXbup6UmF7uLJJHlNs10g6UtIz9QuS5+YVUlwCIuIPkv5QdB2JYXm7xv2bpB8oe8Mbkv5ffq6UbFdJ6q5sYt3++emvRcRrxVWFopR24pTtb0gaIOkISRcp+6VwY0T8stDCCpaPu3xT0mmSzlS2pNDMiPheoYUlrkyThPKhMkcpe5N7v6QPS3pA2c/SxIi4sMDyCmP7rxFxUIMVRCRWDpHE8nZoPttTI2JY0XWgeKUNqVK2RI6yddis7I9r6ZbLaSh/F/tFVbwukq6i9WPDbD9Rv33f5i7frW1vSVtIWiSpd0QszZcX+nvZd21D41jernHMZF9XvprKa8pWgXhvd7Iyrx1bVqXt7s+79/8SEffbHihpoO0OEbGy6NqKlC8J86v8A81XphBfl8/QXmb7+YhYKkkR8a5tlhSyr4uIzzV1roS+X3QBiWIm+7pOyv/9asW5kLRzAbWgQKUNqcp29Tg4f9d6r6Spyn4wTim0qoLkrWPrDVq0jjWpTMstrbDdOSKWSdq3/qTt7mJ/bUnavfIgH9e973ruLY18eTt24loXM9kbiIj+RdeANJQ5pDoiltn+oqT/iYhLbD9ZdFEFOib/t/6da+VM01L/wrQ9SNlSXH+v3A3G9pERcW9++LdCiivGRyPin9J7Le/1Okg6vZiSimf7O8pmJW9pe2n9aUkrlK17WWqN7MT1S9vsxJVt/8lM9gr5+tznSuobEaNsD5A0MCLuLLg0tLLSjkm1/YSySUGXS/piRMyw/XRE7FlwaYVqbGxlmSYFNWT7LGXBfZay8XRnR8Tt+bXSvi5YP9sXRcR3iq4jNezEtX75ZjL1M9kfKftMdtu/k/SYpNMiYo88tD4UEXsXXBpaWZl3QTlb0nck/TEPqDsrm6FcdrZ9YMXBASr3/5MvS9o3Io6X9DFJ59k+O79Wpi5+NN+j+dAHSZLtrW0fX2RBiWAnrvXbQtIbkpZKqrH90YLrKdouEXGJpJWSlA8t4vdtCZW2uz8iJisbl1p/PFfSWcVVlIwvSrq64o/smyrxmn3K/rC+LUkRMS/fsvCWfNwYvzTRmAsqF++PiDfzZbtuK7CmFDS2E9c9BdaTBNsXK3stZmjNmO5Qxd+nElqRrxZSv475LqpYEQLlUdqQmnc1fUvZJIdO9ecj4tDCikpARDwmaa/6kBoRSyqvl23Rekmv2N47Ip6UpIh42/Yxkq6WVOqhIVivxloHS/u7tl6+E9enJR2Un2InrszxysZbEsLWuEDZhOY+tm9Qtnvb5wutCIUo85jU+5StwfYNSWcom/BRGxHfLrSwxJVtHKbt3sqWXFrUyLUDI6JME6bQDLavVtYDcUV+6quSto2IzxdWVAJs95f0ckQsz4+3lNQzIuYVWljBbN+jbJ3Ut5u8uURs91A2TtdinG5plTmkPhYR+9p+qn55JdtTImK/pj63zMq0aD3wfuRrMJ8n6XBl3ZX3S7owIt7Z4Cdu5mxPlXRARKzIjztK+lvZf+fa/oOkvST9WWtvclDq4WcVre4h6a+0updTmbug6hftf9n20ZJekrRtgfW0FeV8VwM0Ux5Gx9jequzBtIH29QFVkiJiRR5Uy25C/oGc7f+WtKvWjF/+d9uHR8RXN/Bp2AyVOaT+KB93+XVJv5TUTdI5xZbUJjBZCNiAfEWMqyR1kdTX9l6S/j0i/qPYygpXa3tEREyQJNvHKdv6stQi4tp86EPfiJhddD2JOFTS4PrtuG1fq2xiGUqmtCG1YlHgJZI+XmQtbQxjMIENu1zSJ5W3jkXENJYUkpSN/b/B9n/lxwsklX2rWNk+VtKlkjpK6m97b0ljI2JEsZUVao6kvpL+kR/3yc+hZEq7Rp3tnW3fYfs126/avj1fK7XUbPe0/et8ML9s1+S7ckmSImJ0cdUBbUNEvNjg1KpCCklIRDwfEftLqpFUExEHRMTz9ddtl3W3su9LGq5ssp3ylUTK/reoq6RZth+0/YCkmZK62Z5gm6ERJVLallRJNyqbffup/HiksvEvHy6sojRcI+k3yrbqk6Rnla2C8OuiCgLamBfzLv+w3UHZxiGzCq4pGRuYxX62pDItb1dvZUQssdcaSbV6fTeXxPlFF4A0lDmkdo6I6yqOr7f9zcKqScd2EXFzvg+5IqLOdulbgYCNcIakX0jaUdJCSfcpW4YKG1bW8e4zbH9WUrt8j/qzJD1UcE2FiohJG7pu++GI+Ehr1YPilC6k2q6fwX+P7TGSblI2Y/0kSXcXVlg63snXp6sfsL6/snG7AJpgu52kX0TEKUXX0gaVdeWQM5X1XP1TWQ/fREk/KrSi9HVq+hZsDkq3TqrtF5T9MmzsXXtERKnHAtneR9lqB3tImi6pWtKJEfFUoYUBbYTtv0o6tHK5JTSNNZgbZ/uXEXFm0XWkpGybypRZ6VpSI6J/c+6zfURE3L+p60lNRDxu+xBJA5UF+dkRsbKJTwOwxlxJf8sneLy3TmpEXFZcSemwfZCyiULTI+K+ikusHNK4A4suAChK6ULqRrhY2U4xpZDv7tGY3WwrIm5t1YKAtuv5/KNK2SzlUrP9aEQMzx9/Wdn43D9KusD2PhHxE4mVQ7BRyjp+uXQIqetXth+CYzdwLSQRUoFmiIgfFF1DYjpUPB4l6YiIqLV9qaRHJP2kmLKQKts9lU08lKSFEfFKg1tKv75uWRBS169Ug3Uj4gtF1wC0ZbZ/HhFfs32HGvn9UeLF2atsb6OsZdkRUStl28fariu2tDahNA0m+UYG/yupu7KVMSSpt+03Jf1HRDwuSRExvaAS0coIqVhLPrP/AkkHKftD+1dlu5+8XmhhQPrql7S7tNAq0tNd0mPKwlbY7hURL9vuohIFsKbY7hwRyxq59ItWL6Y41yjbQvjvlSfzVWZ+I2mvIopCcUo3u1+SbA+SdJwquhMkTYiIWRX33BoR6xunudmyfb+kyZKuz0+dIuljEXF4cVUB2NzY7iypZ0S8UHQtRco3frhKUpeI6Gt7L2VB7T8KLq3V2X4uIgas59qciNi1tWtCsUoXUm1/W9LJytZHXZCf7q1sx6mb6gfxl5Xt6RGxR4NzT0fEnkXVBLQFtp/WBoYJRcSQViwHbYTtv0s6UVlDydD83Dq/h8vA9jhJu0j6raT6rYX7SDpN0gtMriufMnb3f1HS7g2XVbJ9maQZYhD/fbZHSro5Pz5R2eLSADbsmPzf+t2l6rv/T1XJxrhj40TEiw22RS3lLn8RcZbto7RuT+cVEcFmOyVUxpbUZyR9MiL+0eD8TpLui4iBxVSWBttvSdpKa/aOrtKatR4jIroVUhjQRjS2KD2Lj2N9bN8i6TJJ/yXpw5LOljQsIkYWWhiQgDK2pH5N0p9tP6c13Ql9Je0qqfRdCRFR+nUdgQ/Itg+MiL/lBwcoe7MHNOYMZZOjdlTWanif1rTGI2f7yogYVXQdaF2la0mVJNtVynY8qexOmBIRpexiacj2EEn9VPEmhsX8geaxva+kq5XNarekxZL+rX75HACNs73t+i5JmhYRvVuzHhSvlCEV62f7aklDlI3Pre/yj4j4t+KqAtoe290lKSKWFF0L0mX7Ekk/kvSupHuV/f49JyKu3+AnboZsr5L0D629NFnkxztGRMdCCkNhCKlYi+2ZEVFTdB1AW2P71Ii43va5jV2PiMtauyakz/aTEbG37U8pm3x3rqTJEVG6NUHzYXiHRcT8Rq69GBF9CigLBWKcFBp62DYhFdh4W+X/dl3PB9CY+mFVR0v6fclb3n8uaZv1XLukNQtBGmhJxVpsHyJpgqRFkv6pfJcY1ngEgJZn+yeSjlfW3T9c0taS7oyIDxdaWMJsHxER9xddBzY9QirWYnuOsu6mp7VmTKoaLtkFoHG2d1Y2W3t/ZePpHlY2xnBuoYUhWfmEoSURsSrfiatbRCwquq5UsaRbeZRxCSpsWG1ETCi6CKANu1HSFZI+lR+PlPR/ytbABNZi+7SKx5WXftv61bQZbvoWbA4IqWjoCds3SrpDWXe/JJagAjZC54i4ruL4etvfLKwapG6/isedJB0m6XERUjeELuCSIKSioS2VhdNPVJwLSYRUYAMq1ni8x/YYSTcp+9k5SRJbOqJREXFm5bHtrZX93wFKjzGpANACbL+gNWs6NhQRsXMrl4Q2yHYHSdPLukV3vtnO/hHx0AbuuTUiPt2KZaEghFRIkmx/KyIusf1LNdKVEhFnFVAWsNlhZjIq2b5Da37nVkmqkXRzRIwprqpi2X4iIoYWXQeKR3c/6s3K/51aaBXA5u9iSYRU1Lu04nGdpH9ExIKiiknEn22fIOnWoCWt1GhJxXrl3S5dImJp0bUAmwtaibAxbD8cER8puo7WZPstZZtjrFK2fmz9et3dCi0MrY4dp7AW2zfa7mZ7K0nTJc1kZjLQomgZwMboVHQBrS0iukZEVUR0iIhu+TEBtYQIqWioJm85PV7SPZL6S/pcsSUBQGmV7k2NM6faPi8/7mN7eNF1ofURUtFQh3x26fGSJkTESpXwlyTQEmw3ttblvNauA2hj/lvSRyR9Nj9+W9kGGSgZJk6hofHK/ohOkzTZ9k6SGJMKNMF2w53aLOnj+bqXiogR+b8snYONUcbdlT4cEfvYfkKSImKx7Y5FF4XWR0jFWiJinKRx9ce250v6eMXx6RFxbRG1AYnrLWmmpKu0Zr3UYZJ+VmRRSJ/tD0karuz/zZSIWFRxuYzDrVbabqe8F892taTVxZaEItDdjw2KTF3FqbMLKwZI2zBJj0n6nqQlEfGgpHcjYlJETCq0MiTL9pckPSrp05JOlPSI7X+rvx4R04uqrUDjJP1R0va2L5T0V0k/LrYkFIElqLBRWD4H2DDbvSVdLukVSSMiom/BJSFhtmdLOiAiXs+Pe0h6qKw7TtWzPUjSYcp6JP4cEbOa+BRshujux8biXQ2wAflC7P9q+2gxnhtNe13SWxXHb+XnSsf2thWHr0r6v8prEfFG61eFIhFSsbHKOIgf2GgRcZeku4quA2myfW7+cI6kv9u+XVkjwHGSniqssGI9pjXjuftKWpw/3lrSfGVLIqJEGJOKJtn+QsXh3worBAA2H13zj+cl3aY1vVS3S3qhqKKKFBH9I2JnSX+SdGxEbBcRPSQdI+m+YqtDERiTiibZns+4OgBAa7D9dETs2dQ5bP7o7ockyfb6upcsqWdr1gIAZWH7ATUy1j8iDi2gnFS8ZPs/JV2fH58i6aUC60FBCKmo11PSJ5WNAapkSQ+1fjkAUArfqHjcSdIJkurWc29ZnCzpAmXLUEnS5PwcSoaQinp3SuoSEU82vGD7wdYvBwA2fxHxWINTf7P9aCHFJCKfxX+27a7ZYbxddE0oBmNSAQAoSINll6ok7StpXJnXSbW9p6TfSqp/bV6TdHpJNzYoNVpSAQAoTuWyS3XKZvZ/sdCKijde0rkR8YAk2f6YpCslHVBkUWh9hFQAAAoSEaz9ua6t6gOqJEXEg7a3KrIgFIOQCgBAgWwfIKmfKv4mR8RvCyuoeHNtnyfpuvz4VElzC6wHBWFMKgAABbF9naRdJD0paVV+OiLirOKqKpbtbST9QNJByoZC/D9JP4iIhqvPYDNHSAUAoCC2Z0mqCf4YA+tgW1QAAIozXdKHii4iJbbvt711xfE2ticWWROKwZhUAABame07lHVld5U0M18b9Z/11yNiRFG1JWC7iHiz/iAiFtvevsiCUAxCKgAAre/SogtI2GrbfSNiviTZ3kmNbB2LzR8hFQCAVhYRk5pzn+2HI+Ijm7qexHxP0l9tT1K2fuzBkkYVWxKKwMQpAAASZfuJiBhadB2tzfZ2kvbPDx+JiNeKrAfFoCUVAIB0lbUlaQtJbyjLKTW2FRGTC64JrYyQCgAAkmH7YkknSZoh6f+3dzehmtZ1GMe/18Q4L+CMWMKAFtmLggQ1aoYTEZRoUVMxukkzzVoYTApRG9sUtTFyIUnLSlsUiQM6LSyQWqSTSuMY06iRVowLw2Yxiubgy6/FeY5zPM3obM79+595vh84nPu57/vAxWzm4v/2vDa7XYAldc5YUiVJmliSdVV15K3fJCseZjxfAM49wX8fncQ8J1WSpOntgde/cerNXD1BltE8BaztDqF+jqRKkjS9U5JcCWxLsmP5w6raNfu9f/Jk/V4E9iW5jzeeHTu3XxU7ryypkiRN73rgKuA0YPuyZwXsmjzROO6Z/WjOeQSVJElNkuysqtuW3TvR9aonrSQbgHdV1RPdWdTHNamSJPW57hj39kyeYiBJtgP7gHtnnz+UxJHVOeR0vyRJE0uyBTgT2JBkK0d38W8CNrYFG8N3gYuAPwBU1b4k7+kMpB6WVEmSpncZcC1wFnALR0vqc8BNTZlG8XJVHU7ecPrWa8d7WScvS6okSROrqtuB25NcXlV3He+9JNfM3p0nf52dfPC2JO8HbgAeaM6kBm6ckiRpUEn2VtX53TmmlGQj8B3g0tmt3wI/qKqX+lKpgyVVkqRBJXmkqrZ25xhJkh9X1Te6c2jlubtfkqRxOZL0/z7aHUDTsKRKkjSuvPUr0snJkipJ0sSSfCTJptn1hiTfS7I7yc1JNi959f6miFI7S6okSdP7KQvfUQ9wK7AZuHl272eLL1XVzumjDc/R5TnhEVSSJE1vTVW9Mru+cMkO/j8m2dcVaiRJNlbVi8d4dOvkYdTCkVRJkqa3P8lXZtePJrkQIMk5wMt9sfol2ZbkAPD47PMHk/xk8XlV/bwrm6blEVSSJE1stu70VuBjwH+A84GDs58bqurRxnitkjwIXAHcs3j8VpL9VfWB3mSamtP9kiRNrKoOA9fONk+dzcL/x09X1b97k42hqg4u+1rUV7uyqI8lVZKkJlX1HDC3o6bHcTDJNqCSrAVuBB5rzqQGTvdLkqRhJHkHC0shLmFhJ//vgBur6lBrME3OkipJkqThuLtfkiQNI8kPk2xKsjbJfUmeTfKl7lyaniVVkiSN5NLZWt3PAv8E3gd8uzWRWlhSJUnSSBY3dX8GuHN2EoLmkLv7JUnSSH6T5HHgv8DXk5wBvNScSQ3cOCVJkoaS5HTgcFW9mmQjsKmqnunOpWk5kipJkoaR5MtLrpc+umP6NOpkSZUkSSP58JLr9cAngb1YUueO0/2SJGlYSU4DflVVn+rOomm5u1+SJI3sBeDs7hCantP9kiRpGEl2A4vTvGuA84Bf9yVSF6f7JUnSMJJ8fMnHV4B/VdXTXXnUx5IqSZJWjSR7quri7hxaea5JlSRJq8n67gCahiVVkiStJk4BzwlLqiRJkoZjSZUkSatJ3voVnQw8gkqSJA0lyRbgIham9h+uqmeWPL66J5Wm5kiqJEkaRpKvAQ8BO4ArgD8luW7xeVXt78qmaXkElSRJGkaSJ4BtVXVo9vntwANVdW5vMk3NkVRJkjSSQ8DzSz4/P7unOeOaVEmS1C7JN2eXfwceTHI3C2tSPw/8pS2Y2lhSJUnSCE6d/X5y9rPo7oYsGoBrUiVJkjQcR1IlSdIwkvyeY3yrVFV9oiGOGllSJUnSSL615Ho9cDnwSlMWNXK6X5IkDS3JQ1V1UXcOTcuRVEmSNIwkpy/5uAa4ANjcFEeNLKmSJGkkf2ZhTWpYmOb/B/DV1kRq4XS/JEmShuNIqiRJGkqSbcC7WdJTquqOtkBqYUmVJEnDSPIL4L3APuDV2e0CLKlzxul+SZI0jCSPAeeVBWXurekOIEmStMR+YEt3CPVzul+SJLVLspuFaf1TgQNJHgKOLD6vqs91ZVMPS6okSRrBj7oDaCyuSZUkKMFhKgAAAbtJREFUSatGkj1VdXF3Dq0816RKkqTVZH13AE3DkipJklYTp4DnhCVVkiRJw7GkSpKkdknWneirKxpEw7CkSpKkEeyB179x6s1cPUEWDcAjqCRJ0ghOSXIlsC3JjuUPq2rX7Pf+yZOphSVVkiSN4HrgKuA0YPuyZwXsmjyRWnlOqiRJGkaSnVV127J766rqyPH+Ricn16RKkqSRXHeMe3smT6F2TvdLkqR2SbYAZwIbkmzl6C7+TcDGtmBqY0mVJEkjuAy4FjgLuIWjJfU54KamTGrkmlRJkjSMJJdX1V1v8vyaqrp9ykzqYUmVJEmrRpK9VXV+dw6tPDdOSZKk1cRvnJoTllRJkrSaOAU8JyypkiRpNXEkdU5YUiVJUrskNyR55wm8ev+Kh9EQ3DglSZLaJTkMvAA8CfwSuLOqnu1NpU6OpEqSpBE8xcIZqd8HLgAOJLk3yTVJTu2Npg6OpEqSpHbLj5ZKshb4NPBF4JKqOqMtnFpYUiVJUrskj1TV1uM821hVL06dSb0sqZIkqV2Sc6rqb905NA5LqiRJkobjxilJkiQNx5IqSZKk4VhSJUmSNBxLqiRJkobzPylSw2VLu7MGAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        },
        "id": "xASFWi98W0Ve",
        "outputId": "86dfb223-0217-4e32-f353-ec0147d87241"
      },
      "source": [
        "# Sort model results by f1-score\n",
        "all_model_results.sort_values(\"f1\", ascending=False)[\"f1\"].plot(kind=\"bar\", figsize=(10, 7));"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAI9CAYAAAAev/3CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7xmZV3//9ebk4hy0Ji0LwdBI2zygDqiYgfzkJgKnkoIDZOkvoVQlN8wC5XqZ5pZalSSaYopoVIOhqKZh0REBkFlIGpEhcEOAyqYGDD4+f2x1p652eyZvWHde6+193o9H4/9mHsdmP3xdmbP+76ua32uVBWSJEm6e3bouwBJkqTlzDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDnbq6xvvvffedcABB/T17SVJkhbskksuub6qVs11rbcwdcABB7Bu3bq+vr0kSdKCJfnatq45zSdJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6mCnvgvo6oBT/rHvEgD46h8+ve8SJElSDxyZkiRJ6sAwJUmS1MGCwlSSw5NclWRDklPmuL5/ko8nuTTJF5P89PRLlSRJGp55w1SSHYHTgacBq4Gjk6yeddvvAGdX1SOAo4A/n3ahkiRJQ7SQkalDgQ1VdXVV3QqcBRw5654C9mhf7wl8fXolSpIkDddCwtQ+wLUTxxvbc5NeBbwgyUbgPOClc/1GSY5Psi7Juk2bNt2NciVJkoZlWgvQjwb+pqr2BX4aODPJnX7vqjqjqtZU1ZpVq1ZN6VtLkiT1ZyFh6jpgv4njfdtzk44DzgaoqguBXYG9p1GgJEnSkC0kTF0MHJTkwCS70CwwXzvrnmuAJwEk+WGaMOU8niRJWvHmDVNVtRk4ATgfuJLmqb31SU5LckR7228AL0nyBeA9wIuqqharaEmSpKFY0HYyVXUezcLyyXOnTry+Anj8dEuTJEkavmW/N5/ubCj7FYJ7FkqSVj7DlEbDkClJWgzuzSdJktSBI1PSyDliJ0ndODIlSZLUgSNTkjSHoYzYOVonDZ8jU5IkSR0YpiRJkjpwmk+StCBDmfoEpz81LI5MSZIkdWCYkiRJ6sBpPkmSOnD6U45MSZIkdeDIlCRJmroxjdg5MiVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjpYUJhKcniSq5JsSHLKHNf/JMll7de/JfnW9EuVJEkanp3muyHJjsDpwFOAjcDFSdZW1RUz91TVr0/c/1LgEYtQqyRJ0uAsZGTqUGBDVV1dVbcCZwFHbuf+o4H3TKM4SZKkoVtImNoHuHbieGN77k6SPAA4EPjnbVw/Psm6JOs2bdp0V2uVJEkanGkvQD8KeF9V3T7Xxao6o6rWVNWaVatWTflbS5IkLb2FhKnrgP0mjvdtz83lKJzikyRJI7KQMHUxcFCSA5PsQhOY1s6+KcmDgfsAF063REmSpOGaN0xV1WbgBOB84Erg7Kpan+S0JEdM3HoUcFZV1eKUKkmSNDzztkYAqKrzgPNmnTt11vGrpleWJEnS8mAHdEmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6WFCYSnJ4kquSbEhyyjbu+dkkVyRZn+Td0y1TkiRpmHaa74YkOwKnA08BNgIXJ1lbVVdM3HMQ8HLg8VX1zSTfv1gFS5IkDclCRqYOBTZU1dVVdStwFnDkrHteApxeVd8EqKr/nm6ZkiRJw7SQMLUPcO3E8cb23KQfAn4oyQVJPpvk8Ll+oyTHJ1mXZN2mTZvuXsWSJEkDMq0F6DsBBwFPAI4G/irJXrNvqqozqmpNVa1ZtWrVlL61JElSfxYSpq4D9ps43rc9N2kjsLaqbquqrwD/RhOuJEmSVrSFhKmLgYOSHJhkF+AoYO2se/6BZlSKJHvTTPtdPcU6JUmSBmneMFVVm4ETgPOBK4Gzq2p9ktOSHNHedj5wQ5IrgI8DL6uqGxaraEmSpKGYtzUCQFWdB5w369ypE68LOLn9kiRJGg07oEuSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpgwWFqSSHJ7kqyYYkp8xx/UVJNiW5rP36xemXKkmSNDw7zXdDkh2B04GnABuBi5OsraorZt36d1V1wiLUKEmSNFgLGZk6FNhQVVdX1a3AWcCRi1uWJEnS8rCQMLUPcO3E8cb23GzPTfLFJO9Lst9cv1GS45OsS7Ju06ZNd6NcSZKkYZnWAvRzgQOq6mHAR4F3zHVTVZ1RVWuqas2qVaum9K0lSZL6s5AwdR0wOdK0b3tui6q6oapuaQ/fCjxqOuVJkiQN20LC1MXAQUkOTLILcBSwdvKGJD8wcXgEcOX0SpQkSRqueZ/mq6rNSU4Azgd2BN5WVeuTnAasq6q1wIlJjgA2A98AXrSINUuSJA3GvGEKoKrOA86bde7UidcvB14+3dIkSZKGzw7okiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSepgQWEqyeFJrkqyIckp27nvuUkqyZrplShJkjRc84apJDsCpwNPA1YDRydZPcd9uwMnARdNu0hJkqShWsjI1KHAhqq6uqpuBc4Cjpzjvt8DXgv87xTrkyRJGrSFhKl9gGsnjje257ZI8khgv6r6x+39RkmOT7IuybpNmzbd5WIlSZKGpvMC9CQ7AG8AfmO+e6vqjKpaU1VrVq1a1fVbS5Ik9W4hYeo6YL+J433bczN2Bx4CfCLJV4HHAmtdhC5JksZgIWHqYuCgJAcm2QU4Clg7c7GqbqyqvavqgKo6APgscERVrVuUiiVJkgZk3jBVVZuBE4DzgSuBs6tqfZLTkhyx2AVKkiQN2U4LuamqzgPOm3Xu1G3c+4TuZUmSJC0PdkCXJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqYMFhakkhye5KsmGJKfMcf2Xk3wpyWVJPp1k9fRLlSRJGp55w1SSHYHTgacBq4Gj5whL766qh1bVIcDrgDdMvVJJkqQBWsjI1KHAhqq6uqpuBc4Cjpy8oapumji8F1DTK1GSJGm4dlrAPfsA104cbwQeM/umJL8KnAzsAjxxrt8oyfHA8QD777//Xa1VkiRpcKa2AL2qTq+qBwG/BfzONu45o6rWVNWaVatWTetbS5Ik9WYhYeo6YL+J433bc9tyFvCsLkVJkiQtFwsJUxcDByU5MMkuwFHA2skbkhw0cfh04N+nV6IkSdJwzbtmqqo2JzkBOB/YEXhbVa1PchqwrqrWAickeTJwG/BN4NjFLFqSJGkoFrIAnao6Dzhv1rlTJ16fNOW6JEmSlgU7oEuSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpgwWFqSSHJ7kqyYYkp8xx/eQkVyT5YpKPJXnA9EuVJEkannnDVJIdgdOBpwGrgaOTrJ5126XAmqp6GPA+4HXTLlSSJGmIFjIydSiwoaqurqpbgbOAIydvqKqPV9XN7eFngX2nW6YkSdIwLSRM7QNcO3G8sT23LccBH5rrQpLjk6xLsm7Tpk0Lr1KSJGmgproAPckLgDXAH811varOqKo1VbVm1apV0/zWkiRJvdhpAfdcB+w3cbxve+4OkjwZeAXwE1V1y3TKkyRJGraFjExdDByU5MAkuwBHAWsnb0jyCOAtwBFV9d/TL1OSJGmY5g1TVbUZOAE4H7gSOLuq1ic5LckR7W1/BNwbeG+Sy5Ks3cZvJ0mStKIsZJqPqjoPOG/WuVMnXj95ynVJkiQtC3ZAlyRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKmDBYWpJIcnuSrJhiSnzHH9x5N8PsnmJM+bfpmSJEnDNG+YSrIjcDrwNGA1cHSS1bNuuwZ4EfDuaRcoSZI0ZDst4J5DgQ1VdTVAkrOAI4ErZm6oqq+21763CDVKkiQN1kKm+fYBrp043tieu8uSHJ9kXZJ1mzZtuju/hSRJ0qAs6QL0qjqjqtZU1ZpVq1Yt5beWJElaFAsJU9cB+00c79uekyRJGr2FhKmLgYOSHJhkF+AoYO3iliVJkrQ8zBumqmozcAJwPnAlcHZVrU9yWpIjAJI8OslG4GeAtyRZv5hFS5IkDcVCnuajqs4Dzpt17tSJ1xfTTP9JkiSNih3QJUmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktSBYUqSJKkDw5QkSVIHhilJkqQODFOSJEkdGKYkSZI6MExJkiR1YJiSJEnqwDAlSZLUgWFKkiSpA8OUJElSB4YpSZKkDgxTkiRJHRimJEmSOjBMSZIkdWCYkiRJ6sAwJUmS1IFhSpIkqQPDlCRJUgeGKUmSpA4MU5IkSR0YpiRJkjowTEmSJHVgmJIkSerAMCVJktTBgsJUksOTXJVkQ5JT5rh+jyR/116/KMkB0y5UkiRpiOYNU0l2BE4HngasBo5OsnrWbccB36yqHwT+BHjttAuVJEkaooWMTB0KbKiqq6vqVuAs4MhZ9xwJvKN9/T7gSUkyvTIlSZKGKVW1/RuS5wGHV9UvtscvBB5TVSdM3HN5e8/G9vjL7T3Xz/q9jgeObw8PBq6a1v+QjvYGrp/3rvHxfbkz35O5+b7Mzfdlbr4vd+Z7MrchvS8PqKpVc13YaSmrqKozgDOW8nsuRJJ1VbWm7zqGxvflznxP5ub7Mjffl7n5vtyZ78nclsv7spBpvuuA/SaO923PzXlPkp2APYEbplGgJEnSkC0kTF0MHJTkwCS7AEcBa2fdsxY4tn39POCfa775Q0mSpBVg3mm+qtqc5ATgfGBH4G1VtT7JacC6qloL/DVwZpINwDdoAtdyMripx4Hwfbkz35O5+b7Mzfdlbr4vd+Z7Mrdl8b7MuwBdkiRJ22YHdEmSpA4MU5IkSR0YpiRJkjoYZZhKskOSw/quQ5IkLX+jXYCe5NKqekTfdQxVkt2q6ua+6xiKJLsBvwHsX1UvSXIQcHBVfbDn0nqV5D40Pea2PBlcVZ/vr6L+JTmzql4437mxSPKc7V2vqnOWqhYtD0l+fK7zVfWppa5loZa0A/rAfCzJc4Fz7Im1VTti91bg3sD+SR4O/FJV/Uq/lfXu7cAlwOPa4+uA9wKjDVNJfg94EfBlYObvUAFP7KumgfiRyYN2s/hH9VTLEDxzO9cKGGWYSvJttv69uZOq2mMJyxmal0283pVmj+BLGPDPljGPTH0buBdwO/BdIECN/A8wSS6iaby6dmbkLsnlVfWQfivr18yWBpMjmkm+UFUP77u2viS5CnhouwH66CV5OfDbwD2BmVHdALcCZ1TVy/uqTcPVfij5D+BMmj8vxwA/UFWn9lrYgCTZD/jTqnpu37Vsy2hHpqpq975rGKqqujbJ5Knb+6plQG5Nck/aT5JJHgTc0m9Jvbsc2Av4774LGYKqeg3wmiSvMTjNLcnTaUbudp05V1Wn9VfRIBwx60PZXyT5AmCY2moj8MN9F7E9ow1TadLCMcCBVfV7bfL9gar6XM+l9e3adqqvkuwMnARc2XNNQ/BK4MPAfkn+Fng8zRTXmL0GuDTJ5UwEy6o6or+S+ldVL0+yD/AA7riWbLDrPZZCkr8EdgN+kmYpwfOAsf+8BfhOkmOAs2g+rB0NfKffkvqV5M1snQLdATgEGPRazDFP8/0F8D3giVX1w+1C2o9U1aN7Lq1XSfYG3gg8mWbI+SPASVU1+o2rk3wf8Fia9+WzVXV9zyX1Ksl64C3Al2j+LgFQVZ/sragBSPKHNFtqXcHWUd0ae8hM8sWqetjEr/cGPlRVP9Z3bX1KcgDNz9zH0wSIC4Bfq6qv9ldVv5IcO3G4GfhqVV3QVz0LMdqRKeAxVfXIJJcCVNU3242cR60NCMf0XcdA7Qp8k+bvzeokYx9tuLmq3tR3EQP0bJonPcc+DTzbd9tfb07yf4AbgB/osZ5BaEPTkX3XMRTtAxs/VVXL6t+hMYep29r/02bWwKxi4tP1WCV5HfD7ND/4Pgw8DPj1qnpXr4X1LMlrgecD69n656SAMYepf0nyGmAtd5zmG/Rw/BK4GtgZ19TN9sEkewF/RDNlUzTTfZolyaljXUtWVbcneUCSXZbTwy1jnuY7huYfx0cC76CZv/+dqnpvr4X1LMllVXVIkmcDzwBOBj415qfWYMuTaw9ztGGrJB+f43RV1WAfX14KSd4PPBz4GHcMmSf2VtTAJLkHsGtV3dh3LUOU5Jqq2r/vOvqS5J00C87XMrF+rKre0FtR8xjtyFRV/W2SS4An0ayBeVZVudB665+JpwPvraobZz3ZN1aONtzZcVV19eSJJA/sq5gBWdt+aZb24ZYDaH/OtFPl7+y1qJ4kuWlbl2jaa4zZl9uvHYBl8eT96Eamktx3e9er6htLVcsQtYtnn0UzzXcozaPvH6yqx/RaWM8cbbizJJ+vqkfOOndJVY25QSUAbRuN/avqqr5rGYokZwIPAi7jjgvzR/l3KMk1wKOr6r/muHZtVe3XQ1m6m8Y4MnUJzVx9gP1pFhSHJjRcAxzYX2n9q6pT2nVTN7Zz19/BxZHgaMMWSR5M0ytoz1lbhezBRP+gsUryTOD1wC7AgUkOAU4b+9N8wBpgtTtObPFOmvYZdwpTwLuXuJZBSXIud+4OfyOwDnhLVf3v0le1faMbmZqR5K+Av6+q89rjp9FM9f1Sv5X1b/ZQPDDaoXjdWZIjaUYvj+COAfPbwFlV9ZleChuIdvnAE4FPuIvAVkneC5xYVf/Rdy0atiRvBFYB72lPPR+4iSZg7THEfS7HODI147FV9ZKZg6r6UDsiM2rbGoqn+RQ1OknOrqqfTfIl5thHq6oe1kNZvaqqDwAfSPK4qrqw73oG6LY51hqO/klhYG/giiSfwyavW7SjMO8BPlBVo27WOeGwWT0fz01ycVU9uu1vNzhjDlNfT/I7wMwj/8cAX++xnqFwKP6OTmp/fUavVQzTs9sfbLbRuKP1SX4O2DHJQcCJwKhH61qv6ruAgXo9zcjLa5JcTNMJ/YNDnMpaQvdOsn9VXQOQZH/g3u21QbZLGPM0331ptgj58fbUp4BXuwDdoXgtjG005pZkN+AVwE+1p84Hfn/k/zgCkOR+wMyIw+eqyn0dW23fwycCLwEOr6o9ei6pN0l+GvhLmif6QrOW+VeATwAvqao/7a+6uY02TM1IsjvNEyX/03ctQ9D2DjqEZs+s0Q/FJ/k2W6f3ZuZtZh5gqJH/wFtfVT+S5K3A+6rqw0m+MPYwpbkl+Vmahp2foPn782PAy6rqfX3WNQTt05/PZGvvww9W1Uv7rapfbS+yB7eHV01+GEnylKr6aD+VzW20YSrJQ2nWAc20SrgeOLaqLu+vqv4l+Ym5zo99vzXdmW005pbko8DPVNW32uP70CzMf2q/lfUryReAp8yMRrW7TvzT2MN3krNp/v58GPg74JNV5Rq77ZirLUvfxrxm6i3AyVX1cYAkTwDOAA7rs6i+VdUnkzwAOKiq/qmdstix77qGIMmP0rwvb283hN69qr7Sd119sY3GNu09E6Rgy76f399nQQOxw6xpvRtomjKO3V8DR1fV7fPeqRmD6yQ95jB1r5kgBVBVn0hyrz4LGoIkLwGOpxmxexCwD83c9ZP6rKtvSV5Jszj/YODtND2E3kWz0/uozOotNXNu8vCcpatmkL43a/HsA5jjSdAR+nCS87nj4+7n9VjPIFTV+UkOS3IAtqNZqMH9fRpzmLo6ye8CZ7bHL6DZMmTsfpVmyPkigKr6dz9VA/Bs4BE0G7RSVV9v19uN0TO3c60wTP028Okkn2Tr2qDj+y2pf1X1siTPZesHkDOq6u/7rGkIbEezMow5TL0YeDXND/4C/qU9N3a3VNWtMyMNSXZigJ8CenBrVVWSAhjzKGZV/cJC7ktybFW9Y7HrGZIkOwB70iwifmx7+teq6vr+qhqOqno/8P6+6xgY29FMSHIozcM9FydZDRwO/OtMg+3WV3spbjtGuwBdc2vXwHwL+HngpTSPo15RVa/otbCeJflN4CDgKcBraIL3u6vqzb0WNmBDXCS6FJKsq6o1fdcxFEk+XVU/OuvJWPCJWMB2NJPa5RRPoxno+SjwGODjND93z6+qP+ixvO0abZjyiZu5tZ+sj6PpkROaHjlv9VNT8zguE+/L0B7NHZokl85spzIm7VOO19M8mbWlo/XYe9hpbraj2ardaeIQ4B7AfwL7VtVNbeuIi4a848SYp/l84mYO7SO5f9V+qdVO6/1zVX00ycHAwUl2rqrb+q5twMYawJ/f/vqrE+cKeGAPtQxGkjNn76k217kRelXfBQzI5vapxpuTfLmqbgKoqu8mGXS7iDGHKZ+4mbCtvedmDPkTwRL5FPBj7Qjmh2l2L38+zTZEmtvgHl9eClV1YN81DNSPTB606zEf1VMtg9G2o7EzfOPWJLtV1c1M/NlIsicD399yzGHqFfjEzaSZvedmPk1PPuU42pA5IVV1c5LjgL+oqtcluazvovqS5ME0bTMumtw9IMnhVfXh9vCCXorrWdub7WRg/6o6vt2f7+Cq+mDPpfUiyctpnnC8Z5KbZk7T7LF2Rm+FDcQcneHfnGSsneF/vKpugS2zJDN2Bo7tp6SFGe2aKYC28eLMEzef9Ymbude5jHUh8aQkl9Isxv8T4LiqWp/kS1X10J5LW3JJTqQJ3VfSrG84qao+0F7zz0ryd8AlwM9X1UPacPWZqjqk59J6leQ1VfXyvusYGjvDrwxj7z57D+AbwE3A6iQ/Ps/9Y5Akj584OAz/nACcBLwc+Ps2SD2Q5imTMXoJ8KiqehbwBOB3k5zUXhvl1N4sD6qq1wG3AbRTFr4v8Ll2ugaAJHsleVafBQ2EneFXgNFO8yV5Lc2al/VsnYstmrUxY3Yc8LaJH3rfwv5bVNWnmPizUVVXAyf2V1GvdpiZ2quqr7ZbMb2vXXdoaGjWfdyTdno8yYOYeEprxF452aSzqr7VPgr/Dz3WNARzdYb/UI/16G4YbZii2aD14Jn5WTWq6hLg4TNhqqpunLw+xkaMsGXo/f/RLKLddeZ8VT2xt6L6819JDqmqywCq6n+SPAN4GzC6ac85vJLmIYX9kvwtTcfvF/Va0TDMNdoy5n+DgC2d4Z8D/Gh7ys7wy9Bo10wl+RBNn6n/mfdmbTHWNTFJPkLTN+g3gV+mWQy5qap+q9fCepBkX5pHmP9zjmuPr6pRLjyflOT7aNZjBtdjApDkbTQj3ae3p34VuG9Vvai3ogYgyYHAf1TV/7bH9wTuV1Vf7bUw3SVjDlPvBx4OfIw7Nkob69TNgoy4EeMlVfWoJF+caROR5OKqevR8/63GZ2KkoYBPO9KwpVfb7wJPpnlfPgr8QQ47d3oAABOISURBVFV9Z7v/4QqXZB1wWFXd2h7vAlzgz5blZcxDrGvbL90140zf7WJi4D+SPB34OnDfHuvRQCX5c+AH2boG5peSPLmqfnU7/9mK14amU5Lca+wBapadZoIUQLs36i59FqS7brRhqqre0Q6n7l9VV/VdzzIy1gXGv9+uI/sN4M3AHsCv91uSBuqJwA/PbMGU5B00D7qMWvtk8FuBewP7J3k48EtV9Sv9Vta7TUmOqKq1AEmOpNmOSMvIaMNUkmcCrwd2AQ5Mcghw2hj3Q7qLRrkeZqLh4o3AT/ZZiwZvA7A/8LX2eL/23Nj9CfBU2hmBqvqC7WiAZg3m3yb5s/Z4IzD2LXaWnTH3sngVcCjNgkjaJ5NGvXcWQJL7JfnrdoE+SVa3Xb8BqKoT+quuP0kemOTcJNcn+e8kH2h7TUmz7Q5cmeQT7Sa2VwB7JFmbZNRLC6rq2lmnbu+lkAGpqi9X1WOB1cDqqjqsqr48cz3JoDt/qzHakSngtqq6MbnDrNWg9/5ZIn8DvJ1mux2Af6N5iu2v+ypoIN5N8xTSs9vjo2jWxDymt4o0VKf2XcBAXdtO9VWSnWka4V7Zc02DsZ0ny08CRteOZrkZc5han+TngB3bvbNOBD7Tc01DsHdVnd3up0VVbU4y+k+PwG5VdebE8buSvKy3ajRYVfXJ7V1PcmFVPW6p6hmQXwbeSLOn43XAR9i6F6i2bazrVJeVMYepl9KMvtxCM+pwPvD7vVY0DN9pe+TMLJ59LM06oVFKMvPE3oeSnAKcRfPePB84r7fCtJztOv8tK0uSHYE3VtUxfdeyDI31CeplZbR9puaT5M1V9dK+61hqSR5J87TaQ4DLgVXA86rqi70W1pMkX6H5YTbXp8OqKtdN6S4ZcePbTwNPnGwDoPmNtbffcjPmkan5PH7+W1aeqvp8kp8ADqYJEFdV1W3z/GcrVlUduJD7kjylqj662PVIy9jVwAXtIvwtfaaq6g39lTQcSX6U5qGoy6vqIxOXRvkE9XJjmBKwpWPzXH4oCVV1zpIWtPy8lqajszSfsa6B+XL7tQPNE4+jluRzVXVo+/olNOvH/h54ZZJHVtUfwnifoF5uDFOa8cztXCvAMLV9Y/0HUrMkuR/NImuA66rqv2bdMsoeQlX16r5rGJidJ14fDzylqjYleT3wWeAP+ylLd4dhattG9Y9jVf1C3zUscy4+HLm28e9fAnvSPK0GsG+SbwG/UlWfB6iqy3sqsRdJ/rSqfi3Juczx92TEjZJ3SHIfmpG6VNUmaLbdSbK539J0V40+TCXZrapunuPSG5e8mAFon+R7JRObtNJ0hr+h18Kk4fsbmu1RLpo82T4R+3aajdXHaKalyOt7rWJ49gQuofngXkl+oKr+I8m9GdmH+ZVgtE/zTe4TVVXuE9VK8lHgU8C72lPHAE+oqif3V1W/kjwYOJKJqRtgbVVdOXHPOVW1rXVnGoEk/15VB23j2oaq+sGlrknLT5LdgPtV1Vf6rkULN+YwdRHwPJp/FB/Rnru8qh7Sb2X9mus9SPKlqnpoXzX1KclvAUfT9Jfa2J7el6YD+lkzi0SlJG8CHgS8E5jZNmU/4OeBr4x1IXGSL7GdafCqetgSliMtilFP81XVtbO2k7HTN3wkyVHA2e3x82gamo7VccCPzG4PkeQNwHpcJKpWVZ2Y5GnceRTz9Koac4PXZ7S/znQ7n5n2ewGuNdQKMeaRqfcBbwD+jGZ/tZOANVV1VK+F9SzJt4F7sXWfwh3Y2hOmqmqPXgrrSZJ/BZ5aVV+bdf4BwEeq6uB+KpOWl7maT461galWnjGPTLlP1ByqavT9X2b5NeBjSf6drVM3+wM/CIxy2kZ3XZIzqur4vuvoWZI8vqouaA8Oo/mwJi17ox2Z0rYleRhwABNhe8xNO5PsQNOZeHLq5uKqclpYW0zs43inS8AXqmrfpaxnaJI8CngbzVNsAb4JvHimZYS0nI02TCV5Hc3Gxt8FPgw8DPj1qnrXdv/DFS7J22jei/Vsneqrqnpxf1VJw5fkduBr3PGx9pl9Hfepql16KWxgkuwJUFWj3UBdK8+Yw9RlVXVIkmfTLJA8GfhUVY21FwwASa6oqtV91yEtN+1U8JOq6po5rl1bVfv1UFbvkrygqt6V5OS5rrs3n1aCMc9Xz0xhPR14r5+StrgwiWFKuuv+FLjPNq69bikLGZh7tb/uvo0vadkb88jUHwLPopnmOxTYC/hgVT2m18J6luQngLXAfwK30HbntReMNB1JnlJVbootrSCjDVOwZcHojVV1e9t1do+q+s++6+pTkg00U55fYuuaKWa3BpB094y1HUCSB9I8Qf1YmrVkF9KsU72618KkKRhta4QkPz/xevLSO5e+mkHZVFVr+y5CWsHGuu/au4HTgWe3x0cB76Hp8ycta6MNU8CjJ17vCjwJ+DyGqUuTvBs4l2aaDxh3awRpysY6HbBbVZ05cfyuJC/rrRppikYbpqrqpZPHSfai2X9t7O5JE6J+auJcAYYpSXfZRP+tDyU5hebnbAHPB8a8zY5WkFGvmZqUZGfgcrcHkXR3tQ1eH1tVn9nOPedU1XOWsKxeJfkKW/ttzVZV9cAlLkmautGGqSTnsnW4fQdgNXB2VZ3SX1X9SfL/qup1Sd7MHNMQVXViD2VJy85ce9Bpfj7lqOVstNN8wOsnXm8GvlZVG/sqZgCubH9d12sV0vL3sSTPBc6psX5avXteCximtCyNdmRqPkkurKrH9V1Hn9opi3tX1U191yItF0m+TdOo8naaPnYzvdr26LWwgXNET8vZmDugz2fXvgvoQ5J3J9kjyb2Ay4ErfOJGWriq2r2qdqiqnatqj/bYIDU/P9lr2TJMbdtY/2KvbkeingV8CDgQeGG/JUnLRxovSPK77fF+SQ7tuy5Ji8cwpdl2bp9sfBawtqpuY7zBUro7/hx4HPBz7fH/0DSrVCvJXP38vrrUdUjTMuYF6PMZa5fit9D8UPsC8KkkDwBcMyUt3GOq6pFJLgWoqm8m2aXvovqSZPaOCgF+su3tR1Ud0f46mnYRWnlGHaaS3J9mk+MCLp61L98op7aq6k3Am2aOk1wD/OTE8bFV9Y4+apOWiduS7Eg7optkFRP7XI7QvsAVwFvZ2m9qDfDHfRYlTdNop/mS/CLwOeA5wPOAzyZ58cz1qrq8r9qGpBqbJ06d1Fsx0vLwJuDvge9P8gfAp4H/r9+SerUGuAR4Bc3G8p8AvltVn6yqT/ZamTQlo22NkOQq4LCquqE9/j7gM3ZA3z4fX5bml+TBNPt9BvhYVV05z3+y4iXZF/gT4L+AI6pq/55LkqZmzNN8NwDfnjj+dntO2zfO9C3NY2IPOoD/Bt4zea2qvrH0VQ1H2xT5Z5I8HddhaoUZXZhKcnL7cgNwUZIP0ASEI4Ev9lbY8jHWhfnSfC5h65qg/YFvtq/3Aq6haTMyelX1j8A/9l2HNE1jXDO1e/v1ZeAf2DrS8gHgK30VNWRJfmHi8ILeCpEGrKoObDft/SfgmVW1d1V9H/AM4CP9VidpMY12zZQWLsk1rm+QFibJl6rqofOdk7RyjG6ab0aSjzPH+p+qemIP5fQuybamOAPcbylrkZa5ryf5HeBd7fExwNd7rEfSIhttmAJ+c+L1rsBzgc3buHcM7gc8lWadx6QAn1n6cqRl62jglTTtEQA+1Z6TtEKNNkxV1SWzTl2Q5HO9FDMMHwTuXVWXzb6Q5BNLX460PLVP7Z2UZPfmsP6n75okLa7Rrpma9RjzDsCjgDfZZ0pSF0keCrwTmPkZcz1wrI2ApZVrtCNT3PEx5s00T/Id12tFklaCtwAnV9XHAZI8ATgDOKzPoiQtntGGqaqy54ukxXCvmSAFUFWfSHKvPguStLhGG6YAkhwGHMDE+1BV7+ytIEkrwdVJfhc4sz1+AXB1j/VIWmRjXjN1JvAg4DLg9vZ0VdWJ/VUlablLch/g1cCP0iwl+Bfg1VU1+0lZSSvEmMPUlcDqGusbIEmSpmKM28nMuBy4f99FSFpZknw0yV4Tx/dJcn6fNUlaXKNbM5XkXJqh992BK9reUrfMXK+qI/qqTdKKsHdVfWvmoKq+meT7+yxI0uIaXZgCXt93AZJWtO8l2b+qrgFI8gDm2LpK0soxujBVVZ9cyH1JLqyqxy12PZJWnFcAn07ySZo+dj8GHN9vSZIW02gXoM8nyaVV9Yi+65C0/CTZG3hse/jZqrq+z3okLa7RjUzdBaZMSXfXPYBv0PyMXZ2EqvpUzzVJWiSGKUmaoiSvBZ4PrAe+154uwDAlrVCjC1NJ7lFVt8x/J1n0YiStRM8CDl7gzxlJK8AY+0xdCFs6oG/PC5egFkkrz9XAzn0XIWnpjG5kCtglyc8BhyV5zuyLVXVO++vlS16ZpJXgZuCyJB/jjj3s3KpKWqHGGKZ+GTgG2At45qxrBZyz5BVJWknWtl+SRmK0rRGSnFBVfzbr3ELXU0nSNiW5J7B/VV3Vdy2SFt8Y10zNePEc5y5c8iokrShJnglcBny4PT4kiSNV0go2umm+JPcH9gHumeQRbH1qbw9gt94Kk7RSvAo4FPgEQFVdluSBfRYkaXGNLkwBTwVeBOwL/DFbw9RNwG/3VJOkleO2qroxuUN3le9t62ZJy9/owlRVvQN4R5LnVtX7t3VfkmPbeyXprljfPjG8Y5KDgBOBz/Rck6RFNNoF6PNJ8vmqemTfdUhaXpLsRrPZ8U+1p84Hfr+q/re/qiQtJsPUNrjRsaTFkOTNVfXSvuuQND1jfppvPqZMSYvh8X0XIGm6DFPb5t58kiRpXqMLU0kek2SP9vU9k7w6yblJXptkz4lbL+ipREmStIyMLkwBb6PZOwvgjcCewGvbc2+fuamqTlj60iSNgKPe0gozutYIwA5Vtbl9vWbiib1PJ7msr6IkrSxJdquqm+e49MYlL0bSohrjyNTlSX6hff2FJGsAkvwQcFt/ZUlaCZIcluQK4F/b44cn+fOZ61X1N33VJmlxjK41Qrsu6o3AjwHXA48Erm2/TqyqL/RYnqRlLslFwPOAtTPtVZJcXlUP6bcySYtldNN8VXUj8KJ2EfqBNO/Bxqr6r34rk7RSVNW1s7aTub2vWiQtvtGFqRlVdRPgKJSkabs2yWFAJdkZOAm4sueaJC2i0U3zSdJiSrI3zVKCJ9M8ufcR4KSquqHXwiQtGsOUJElSB2N8mk+SFk2S1yXZI8nOST6WZFOSF/Rdl6TFY5iSpOn6qXZN5jOArwI/CLys14okLSrDlCRN18yDPU8H3ts+QSxpBRvt03yStEg+mORfge8C/zfJKuB/e65J0iJyAbokTVmS+wI3VtXtSXYD9qiq/+y7LkmLw5EpSZqiJD8/8Xry0juXvhpJS8EwJUnT9eiJ17sCTwI+j2FKWrGc5pOkRZRkL+Csqjq871okLQ6f5pOkxfUdmn1AJa1QTvNJ0hQlOReYGfLfAVgNnN1fRZIWm9N8kjRFSX5i4nAz8LWq2thXPZIWn2FKkpZQkgur6nF91yFpelwzJUlLa9e+C5A0XYYpSVpaTgdIK4xhSpIkqQPDlCQtrcx/i6TlxNYIkjRlSe4PHEozpXfxrH35XthPVZIWiyNTkjRFSX4R+BzwHOB5wGeTvHjmelVd3ldtkhaHrREkaYqSXAUcVlU3tMffB3ymqg7utzJJi8WRKUmarhuAb08cf7s9J2mFcs2UJE1BkpPblxuAi5J8gGbN1JHAF3srTNKiM0xJ0nTs3v765fZrxgd6qEXSEnLNlCRJUgeOTEnSFCX5OHN0Oa+qJ/ZQjqQlYJiSpOn6zYnXuwLPBTb3VIukJeA0nyQtsiSfq6pD+65D0uJwZEqSpijJfScOdwAeBezZUzmSloBhSpKm6xKaNVOhmd77CnBcrxVJWlRO80mSJHXgyJQkTVmSw4ADmPgZW1Xv7K0gSYvKMCVJU5TkTOBBwGXA7e3pAgxT0grlNJ8kTVGSK4HV5Q9XaTTc6FiSputy4P59FyFp6TjNJ0lTkORcmum83YErknwOuGXmelUd0VdtkhaXYUqSpuP1fRcgqR+umZKkJZTkwqp6XN91SJoe10xJ0tLate8CJE2XYUqSlpbTAdIKY5iSJEnqwDAlSVOQ5B4LvXVRC5G05AxTkjQdF8KWDujb88IlqEXSErI1giRNxy5Jfg44LMlzZl+sqnPaXy9f8sokLSrDlCRNxy8DxwB7Ac+cda2Ac5a8IklLwj5TkjRFSU6oqj+bde4eVXXLtv4bScuba6YkabpePMe5C5e8CklLxmk+SZqCJPcH9gHumeQRbH1qbw9gt94Kk7ToDFOSNB1PBV4E7Av8MVvD1E3Ab/dUk6Ql4JopSZqiJM+tqvdv5/qxVfWOpaxJ0uIyTEnSEkry+ap6ZN91SJoeF6BL0tKyA7q0whimJGlpOR0grTCGKUlaWo5MSSuMYUqSpiDJiUn2W8CtFyx6MZKWlAvQJWkKktwIfAf4MvAe4L1VtanfqiQtBUemJGk6rqbpMfV7wKOAK5J8OMmxSXbvtzRJi8mRKUmagtktD5LsDDwNOBp4clWt6q04SYvKMCVJU5Dk0qp6xDau7VZVNy91TZKWhmFKkqYgyQ9V1b/1XYekpWeYkiRJ6sAF6JIkSR0YpiRJkjowTEmSJHVgmJIkSerg/wfKOA/QnS5/6AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8v7tXJ1XrWl"
      },
      "source": [
        "## Uploading our model training logs to TesnorBoard.dev\n",
        "We can further inspect our model's performance using TensorBoard.dev: https://tensorboard.dev/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iN5nWpwqYNNt"
      },
      "source": [
        "# # View TensorBoard logs of transfer learning modelling experiments (plus all of our other models)\n",
        "# # Upload TensorBoard dev records\n",
        "# !tensorboard dev upload --logdir ./model_logs/ \\\n",
        "#   --name \"NLP Modelling Experiments ZTM TF Course Video\" \\\n",
        "#   --description \"Comparing multiple different types of model architectures on the Kaggle Tweets text classification dataset\" \\\n",
        "#   --one_shot # exit the uploader once uploading is finished"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eE9WoH_nZsMb"
      },
      "source": [
        "Now I've ran the cell above, my modelling experiments are visable on TensorBoard.dev: https://tensorboard.dev/experiment/Eacboed3RbKPWIGcXe1Z6g/\n",
        "\n",
        "> 📖 **Resource:** TensorBoard is great for quickly tracking experiments but for larger scale experiments and a whole bunch more tracking options, check out Weights & Biases: https://wandb.ai/site"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OEVX02N5aiyp"
      },
      "source": [
        "# # See the previous TensorBoard Dev experiments you've run...\n",
        "# !tensorboard dev list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cXTfCURJZbts"
      },
      "source": [
        "# If you need to delete an experiment from TensorBoard, you can run the following:\n",
        "# !tensorboard dev delete --experiment_id Eacboed3RbKPWIGcXe1Z6gB"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iV-oY5X6bmCh"
      },
      "source": [
        "## Saving and loading a trained model \n",
        "\n",
        "There are two main formats to save a model to in TensorFlow:\n",
        "1. The HDF5 format\n",
        "2. The `SavedModel` format (this is the default when using TensorFlow)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8NuJcP0G-d_0"
      },
      "source": [
        "# Save TF Hub Sentence Encoder model to HDF5 format\n",
        "model_6.save(\"model_6.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "83dzH-v6-vPP"
      },
      "source": [
        "# Load model with custom Hub Layer (required HDF5 format)\n",
        "import tensorflow_hub as hub\n",
        "loaded_model_6 = tf.keras.models.load_model(\"model_6.h5\",\n",
        "                                            custom_objects={\"KerasLayer\": hub.KerasLayer})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6zSLAatG_Xg0",
        "outputId": "971c0149-62cd-403d-81f1-139d549e88aa"
      },
      "source": [
        "# How does our loaded model perform?\n",
        "loaded_model_6.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "24/24 [==============================] - 1s 14ms/step - loss: 0.4289 - accuracy: 0.8150\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4289097487926483, 0.8149606585502625]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iUXdwTKJ_mmC"
      },
      "source": [
        "Now let's save to the `SavedModel` format... (see more on this here: https://www.tensorflow.org/tutorials/keras/save_and_load)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KH56jDFz_2T5",
        "outputId": "bfd21ffe-af2d-4ec2-c0df-7dc5a65b9d4a"
      },
      "source": [
        "# Save TF Hub Sentence Ecnoder model to SavedModel format (default)\n",
        "model_6.save(\"model_6_SavedModel_format\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: model_6_SavedModel_format/assets\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: model_6_SavedModel_format/assets\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8KncUg_M_9gz",
        "outputId": "19e677b4-a123-475e-d104-e56b0418a82b"
      },
      "source": [
        "# Load in a model from the SavedModel format\n",
        "loaded_model_6_SavedModel_format = tf.keras.models.load_model(\"model_6_SavedModel_format\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fea32001950> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fea32001950> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YPDGs88WAWaP",
        "outputId": "5be41b6e-e202-4585-bb88-8f6eaf23c3f5"
      },
      "source": [
        "# Evaluate model in SavedModel format\n",
        "loaded_model_6_SavedModel_format.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "24/24 [==============================] - 1s 12ms/step - loss: 0.4289 - accuracy: 0.8150\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4289097785949707, 0.8149606585502625]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sPHYDcL-AbzG"
      },
      "source": [
        "## Finding the most wrong examples\n",
        "\n",
        "* If our best model still isn't perfect, what examples is it getting wrong?\n",
        "* And of these wrong examples which ones is it getting *most* wrong (those will predicition probabilities closest to the opposite class)\n",
        "\n",
        "For example if a sample should have a label of 0 but our model predicts a prediction probability of 0.999 (really close to 1) and vice versa."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XgSL1VH2CHBu",
        "outputId": "601f485a-39d8-4e40-d2d0-021a6627f3f2"
      },
      "source": [
        "# Download a pretrained model from Google Storage\n",
        "!wget https://storage.googleapis.com/ztm_tf_course/08_model_6_USE_feature_extractor.zip\n",
        "!unzip 08_model_6_USE_feature_extractor.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-14 06:19:57--  https://storage.googleapis.com/ztm_tf_course/08_model_6_USE_feature_extractor.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.133.128, 74.125.140.128, 108.177.15.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.133.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 960779165 (916M) [application/zip]\n",
            "Saving to: ‘08_model_6_USE_feature_extractor.zip’\n",
            "\n",
            "08_model_6_USE_feat 100%[===================>] 916.27M  46.6MB/s    in 25s     \n",
            "\n",
            "2021-04-14 06:20:23 (36.8 MB/s) - ‘08_model_6_USE_feature_extractor.zip’ saved [960779165/960779165]\n",
            "\n",
            "Archive:  08_model_6_USE_feature_extractor.zip\n",
            "   creating: 08_model_6_USE_feature_extractor/\n",
            "   creating: 08_model_6_USE_feature_extractor/assets/\n",
            "   creating: 08_model_6_USE_feature_extractor/variables/\n",
            "  inflating: 08_model_6_USE_feature_extractor/variables/variables.data-00000-of-00001  \n",
            "  inflating: 08_model_6_USE_feature_extractor/variables/variables.index  \n",
            "  inflating: 08_model_6_USE_feature_extractor/saved_model.pb  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9OTLTVMSCkbL",
        "outputId": "dd15fa49-c139-400c-ea49-52f485945e3e"
      },
      "source": [
        "# Import previously trained model from Google Storage\n",
        "model_6_pretrained = tf.keras.models.load_model(\"08_model_6_USE_feature_extractor\")\n",
        "model_6_pretrained.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fea3032e7a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fea3032e7a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "24/24 [==============================] - 1s 12ms/step - loss: 0.4272 - accuracy: 0.8163\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.42723122239112854, 0.8162729740142822]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4ML3jOCvCzW8",
        "outputId": "446da531-9ead-4671-a990-efcdb9207296"
      },
      "source": [
        "# Make predictions with the loaded model from GS\n",
        "model_6_pretrained_pred_probs = model_6_pretrained.predict(val_sentences)\n",
        "model_6_pretrained_preds = tf.squeeze(tf.round(model_6_pretrained_pred_probs))\n",
        "model_6_pretrained_preds[:10] # these should be in label format"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "nwa_Bk9nB4Tl",
        "outputId": "423ea869-96d0-4d5e-872b-941e9f203a81"
      },
      "source": [
        "# Create DataFrame with validation sentences, validation labels and best performing model prediction labels + probabilities\n",
        "val_df = pd.DataFrame({\"text\": val_sentences,\n",
        "                       \"target\": val_labels,\n",
        "                       \"pred\": model_6_pretrained_preds,\n",
        "                       \"pred_prob\": tf.squeeze(model_6_pretrained_pred_probs)})\n",
        "val_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>DFR EP016 Monthly Meltdown - On Dnbheaven 2015...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.159757</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>FedEx no longer to transport bioterror germs i...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.747162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Gunmen kill four in El Salvador bus attack: Su...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.988749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@camilacabello97 Internally and externally scr...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.196229</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Radiation emergency #preparedness starts with ...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.707808</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target  pred  pred_prob\n",
              "0  DFR EP016 Monthly Meltdown - On Dnbheaven 2015...       0   0.0   0.159757\n",
              "1  FedEx no longer to transport bioterror germs i...       0   1.0   0.747162\n",
              "2  Gunmen kill four in El Salvador bus attack: Su...       1   1.0   0.988749\n",
              "3  @camilacabello97 Internally and externally scr...       1   0.0   0.196229\n",
              "4  Radiation emergency #preparedness starts with ...       1   1.0   0.707808"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 355
        },
        "id": "inTan9xLDaj1",
        "outputId": "979772a5-9093-4d71-cc4a-fe08cce1ef88"
      },
      "source": [
        "# Find the wrong predictions and sort by prediction probabilities\n",
        "most_wrong = val_df[val_df[\"target\"] != val_df[\"pred\"]].sort_values(\"pred_prob\", ascending=False)\n",
        "most_wrong[:10] # these are false positives"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>? High Skies - Burning Buildings ? http://t.co...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.910196</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>759</th>\n",
              "      <td>FedEx will no longer transport bioterror patho...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.876982</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>628</th>\n",
              "      <td>@noah_anyname That's where the concentration c...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.852300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>209</th>\n",
              "      <td>Ashes 2015: AustraliaÛªs collapse at Trent Br...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.835454</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>251</th>\n",
              "      <td>@AshGhebranious civil rights continued in the ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.827213</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>393</th>\n",
              "      <td>@SonofLiberty357 all illuminated by the bright...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.814816</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.810840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>@madonnamking RSPCA site multiple 7 story high...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.803122</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>119</th>\n",
              "      <td>@freefromwolves GodsLove &amp;amp; #thankU brother...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.766901</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>344</th>\n",
              "      <td>Air Group is here to the rescue! We have 24/7 ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.766625</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                  text  target  pred  pred_prob\n",
              "31   ? High Skies - Burning Buildings ? http://t.co...       0   1.0   0.910196\n",
              "759  FedEx will no longer transport bioterror patho...       0   1.0   0.876982\n",
              "628  @noah_anyname That's where the concentration c...       0   1.0   0.852300\n",
              "209  Ashes 2015: AustraliaÛªs collapse at Trent Br...       0   1.0   0.835454\n",
              "251  @AshGhebranious civil rights continued in the ...       0   1.0   0.827213\n",
              "393  @SonofLiberty357 all illuminated by the bright...       0   1.0   0.814816\n",
              "109  [55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...       0   1.0   0.810840\n",
              "49   @madonnamking RSPCA site multiple 7 story high...       0   1.0   0.803122\n",
              "119  @freefromwolves GodsLove &amp; #thankU brother...       0   1.0   0.766901\n",
              "344  Air Group is here to the rescue! We have 24/7 ...       0   1.0   0.766625"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 117
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "PQV_SFoSESI1",
        "outputId": "8af9a512-1cdc-4317-e089-d3e4f88e649f"
      },
      "source": [
        "most_wrong.tail() # these are false negatives"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>411</th>\n",
              "      <td>@SoonerMagic_ I mean I'm a fan but I don't nee...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.043918</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>233</th>\n",
              "      <td>I get to smoke my shit in peace</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.042087</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Why are you deluged with low self-image? Take ...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038998</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>244</th>\n",
              "      <td>Reddit Will Now QuarantineÛ_ http://t.co/pkUA...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038949</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Ron &amp;amp; Fez - Dave's High School Crush https...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.037186</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                  text  target  pred  pred_prob\n",
              "411  @SoonerMagic_ I mean I'm a fan but I don't nee...       1   0.0   0.043918\n",
              "233                    I get to smoke my shit in peace       1   0.0   0.042087\n",
              "38   Why are you deluged with low self-image? Take ...       1   0.0   0.038998\n",
              "244  Reddit Will Now QuarantineÛ_ http://t.co/pkUA...       1   0.0   0.038949\n",
              "23   Ron &amp; Fez - Dave's High School Crush https...       1   0.0   0.037186"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_1Me95bIEH75"
      },
      "source": [
        "Let's remind ourselves of the target labels...\n",
        "* `0` = not diaster\n",
        "* `1` = diaster"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UhUESFzvDhqF",
        "outputId": "2fa10a18-25fa-42d5-e72d-968e02e79b06"
      },
      "source": [
        "# Check the false positives (model predicted 1 when should've been 0)\n",
        "for row in most_wrong[:10].itertuples():\n",
        "  _, text, target, pred, pred_prob = row\n",
        "  print(f\"Target: {target}, Pred: {pred}, Prob: {pred_prob}\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Target: 0, Pred: 1.0, Prob: 0.9101957678794861\n",
            "Text:\n",
            "? High Skies - Burning Buildings ? http://t.co/uVq41i3Kx2 #nowplaying\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.8769820928573608\n",
            "Text:\n",
            "FedEx will no longer transport bioterror pathogens in wake of anthrax lab mishaps http://t.co/lHpgxc4b8J\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.8523000478744507\n",
            "Text:\n",
            "@noah_anyname That's where the concentration camps and mass murder come in. \n",
            " \n",
            "EVERY. FUCKING. TIME.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.8354544043540955\n",
            "Text:\n",
            "Ashes 2015: AustraliaÛªs collapse at Trent Bridge among worst in history: England bundled out Australia for 60 ... http://t.co/t5TrhjUAU0\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.8272132873535156\n",
            "Text:\n",
            "@AshGhebranious civil rights continued in the 60s. And what about trans-generational trauma? if anything we should listen to the Americans.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.814815878868103\n",
            "Text:\n",
            "@SonofLiberty357 all illuminated by the brightly burning buildings all around the town!\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.8108396530151367\n",
            "Text:\n",
            "[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES WITH MAGNE-TRACTION INSTRUCTIONS http://t.co/xEZBs3sq0y http://t.co/C2x0QoKGlY\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.80312180519104\n",
            "Text:\n",
            "@madonnamking RSPCA site multiple 7 story high rise buildings next to low density character residential in an area that floods\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.7669008374214172\n",
            "Text:\n",
            "@freefromwolves GodsLove &amp; #thankU brother Danny for RT of NEW VIDEO http://t.co/cybKsXHF7d The Coming Apocalyptic US Earthquake &amp; Tsunami\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1.0, Prob: 0.7666252255439758\n",
            "Text:\n",
            "Air Group is here to the rescue! We have 24/7 Emergency Service! Learn more about it here - http://t.co/9lyx7zMtHE http://t.co/5PbC96rTMJ\n",
            "\n",
            "----\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-T4lJhV6Fcxa",
        "outputId": "0a4a7b68-9351-42ee-a95f-75d4100d2078"
      },
      "source": [
        "# Check the false negatives (model predicted 0 when should've been 1)\n",
        "for row in most_wrong[-10:].itertuples():\n",
        "  _, text, target, pred, pred_prob = row\n",
        "  print(f\"Target: {target}, Pred: {pred}, Prob: {pred_prob}\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Target: 1, Pred: 0.0, Prob: 0.06730344891548157\n",
            "Text:\n",
            "@DavidVonderhaar At least you were sincere ??\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.05507582053542137\n",
            "Text:\n",
            "@willienelson We need help! Horses will die!Please RT &amp; sign petition!Take a stand &amp; be a voice for them! #gilbert23 https://t.co/e8dl1lNCVu\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.05460337549448013\n",
            "Text:\n",
            "Lucas Duda is Ghost Rider. Not the Nic Cage version but an actual 'engulfed in flames' badass. #Mets\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.054596975445747375\n",
            "Text:\n",
            "going to redo my nails and watch behind the scenes of desolation of smaug ayyy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.04963727295398712\n",
            "Text:\n",
            "You can never escape me. Bullets don't harm me. Nothing harms me. But I know pain. I know pain. Sometimes I share it. With someone like you.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.043918490409851074\n",
            "Text:\n",
            "@SoonerMagic_ I mean I'm a fan but I don't need a girl sounding off like a damn siren\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.04208682104945183\n",
            "Text:\n",
            "I get to smoke my shit in peace\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.03899793699383736\n",
            "Text:\n",
            "Why are you deluged with low self-image? Take the quiz: http://t.co/XsPqdOrIqj http://t.co/CQYvFR4UCy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.03894946351647377\n",
            "Text:\n",
            "Reddit Will Now QuarantineÛ_ http://t.co/pkUAMXw6pm #onlinecommunities #reddit #amageddon #freespeech #Business http://t.co/PAWvNJ4sAP\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0.0, Prob: 0.03718579187989235\n",
            "Text:\n",
            "Ron &amp; Fez - Dave's High School Crush https://t.co/aN3W16c8F6 via @YouTube\n",
            "\n",
            "----\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Myh5WDGiGC_J"
      },
      "source": [
        "## Making predictions on the test dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h_adHc7tGh6z",
        "outputId": "4be5873c-870a-4fef-a294-f2fdd16f9f26"
      },
      "source": [
        "# Making predictions on the test dataset and visualizing them\n",
        "test_sentences = test_df[\"text\"].to_list()\n",
        "test_samples = random.sample(test_sentences, 10)\n",
        "for test_sample in test_samples:\n",
        "  pred_prob = tf.squeeze(model_6_pretrained.predict([test_sample])) # our model expects a list as input\n",
        "  pred = tf.round(pred_prob)\n",
        "  print(f\"Pred: {int(pred)}, Prob: {pred_prob}\")\n",
        "  print(f\"Text:\\n{test_sample}\\n\")\n",
        "  print(\"-----\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Pred: 0, Prob: 0.17732585966587067\n",
            "Text:\n",
            "@LauraE303B @SheilaGunnReid A war we'll never win? There r 2500 Yezidi women trapped as slaves.We know where they are. Doing nothing is evil\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 0, Prob: 0.04374966397881508\n",
            "Text:\n",
            "(rubs lamp greets genie) just need the 1 wish thanks. pls obliterate anyone who has used the hashtag #youcantsitwithus\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 1, Prob: 0.9137933850288391\n",
            "Text:\n",
            "Horrible Accident Man Died In Wings of Airplane (29-07-2015) http://t.co/hG8u2kR1Rq\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 1, Prob: 0.9550248384475708\n",
            "Text:\n",
            "#USGS M 1.9 - 5km S of Volcano Hawaii: Time2015-08-06 01:04:01 UTC2015-08-05 15:04:01 -10:00 at epicenter... http://t.co/meVvkaXcdE #SM\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 0, Prob: 0.06789832562208176\n",
            "Text:\n",
            "@atlwtmgc damn this blew up\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 1, Prob: 0.9011690020561218\n",
            "Text:\n",
            "Shot 12 times. Found dead in cuffs after being involved in a car accident. Officers told ambulance not to treat him. https://t.co/MEUDJwaaNg\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 1, Prob: 0.9532992243766785\n",
            "Text:\n",
            "Obama Declares Disaster for Typhoon-Devastated Saipan: Obama signs disaster declaration for Northern Marians a... http://t.co/AslUFEKOXN\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 0, Prob: 0.05157076194882393\n",
            "Text:\n",
            "I added a video to a @YouTube playlist http://t.co/DzHDBiajS5 Panic! At The Disco: The Ballad Of Mona Lisa [OFFICIAL VIDEO]\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 0, Prob: 0.35549232363700867\n",
            "Text:\n",
            "This fly is a major problem for us: It will ruin our batch and we need to destroy it and every trace of it so we can cook.\n",
            "\n",
            "-----\n",
            "\n",
            "Pred: 0, Prob: 0.07776769250631332\n",
            "Text:\n",
            "Paul Hollywood 'what happened....looks like a mudslide... It's like chewing on a rubber tyre... It looks a mess...' Oh dear Dorrett! #GBBO\n",
            "\n",
            "-----\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1ARtfoAHQf_"
      },
      "source": [
        "## Your challenge... predicting on Tweets from the wild\n",
        "\n",
        "Go to your favourite Twitter account and copy one of their latest Tweets.\n",
        "\n",
        "Then pass that Tweet through our trained model.\n",
        "\n",
        "Is that Tweet a disaster or not disaster (according to the model)? Is the model right or wrong?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VN-XJuilI1ls"
      },
      "source": [
        "## The speed/score tradeoff"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AIJrWpSfJ9X0"
      },
      "source": [
        "# Let's make a function to measure the time of prediction\n",
        "import time\n",
        "def pred_timer(model, samples):\n",
        "  \"\"\"\n",
        "  Times how long a model takes to make predictions on samples.\n",
        "  \"\"\"\n",
        "  start_time = time.perf_counter() # get start time\n",
        "  model.predict(samples) # make predictions\n",
        "  end_time = time.perf_counter() # get finish time\n",
        "  total_time = end_time-start_time # calculuate how long predictons took to make\n",
        "  time_per_pred = total_time/len(samples)\n",
        "  return total_time, time_per_pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8NjtO6ZnKAk8",
        "outputId": "4dfd5e5e-e814-4802-a39a-b8b6f572284c"
      },
      "source": [
        "# Calculate TF Hub Sentence Encoder time per pred\n",
        "model_6_total_pred_time, model_6_time_per_pred = pred_timer(model=model_6_pretrained,\n",
        "                                                            samples=val_sentences)\n",
        "model_6_total_pred_time, model_6_time_per_pred"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.313799298000049, 0.0004118101023622691)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QviAtHxrKBYL",
        "outputId": "6c1f4e96-275a-412d-ce34-0abb5d9b4634"
      },
      "source": [
        "# Calculate our baseline model times per pred\n",
        "baseline_total_pred_time, baseline_time_per_pred = pred_timer(model_0, val_sentences)\n",
        "baseline_total_pred_time, baseline_time_per_pred"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.024128293999638117, 3.16644278210474e-05)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 134
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XpVh0HV7LsAc",
        "outputId": "d6884e12-6891-45bf-bc40-c981954beaed"
      },
      "source": [
        "# Get results for pretrained GS model\n",
        "model_6_pretrained_results = calculate_results(y_true=val_labels,\n",
        "                                               y_pred=model_6_pretrained_preds)\n",
        "model_6_pretrained_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 81.62729658792651,\n",
              " 'f1': 0.8148082644367335,\n",
              " 'precision': 0.818446310697231,\n",
              " 'recall': 0.8162729658792651}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 135
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "id": "3JHbdjJiLaUP",
        "outputId": "02ae01e0-70fc-4980-89fb-aaa4cbee2fac"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.scatter(baseline_time_per_pred, baseline_results[\"f1\"], label=\"baseline\")\n",
        "plt.scatter(model_6_time_per_pred, model_6_pretrained_results[\"f1\"], label=\"tf_hub_sentence_encoder\")\n",
        "plt.legend()\n",
        "plt.title(\"F1-score versus time per prediction\")\n",
        "plt.xlabel(\"Time per prediction\")\n",
        "plt.ylabel(\"F1-score\");"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAG5CAYAAAA3e7gZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dfbxVZZ3//9dHQDHLO6T5pqhgKcrNkZsj3lWiZlg6aqWGaZM3ZWZm32aidMoyJ7+j2S9nNEytURpL0bSM1IJJMbVMPQyKoqKoJKAVEqgQKODn98de57A5njuEfc5ZnNfz8dgP1r7WdV3rWtfedt6tm70iM5EkSVL3t1lXD0CSJEkdY3CTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEnSeoiIf42IH3X1OLq7iBgbEQuq3s+OiLFvoZ/3RcScjTo4qcQMblI3FBHzImJFRCyreu1YrLs6IuZExBsRcXIXD3WT1jx8AGTm/8vMT3fVmMoqM4dm5t3t1YuIjIj3VLW7NzMH13RwUokY3KTu6x8z8+1VrxeK8keAM4H/7cKxARARvXvitstmY8xVRPTaGGORtGEMblLJZObEzLwTWNle3YjoGxE/iYjFEbE0Ih6KiH8o1m0fEddGxAsRsSQibq1q95mImBsRf4uIKY1H+4p1GRGfj4ingaeLsiMj4uFiG3+IiLpWxvODiPhus7JfRsQ/F8s7RsQtEbEoIp6LiLOr6p0fETcX+/MKcHJEjImIhoh4JSL+EhHfK+q+6UhZcRTzA8Vyi+2a1d8K+DWwY/VRz2IcPynqDCzm45SImF/M4xkRsU9EzCrm4/vN+j01Ip4o6k6NiF1bmavGvk8vPqMXI+LLVes3i4hzIuKZ4vO9KSK2b9b2tIh4Hrirhf7HRsSC4tTvS8X8nFi1flLxed0REcuBg9v5fLYs2iyJiMeBfdqY/17Fdp+JiFcjYkZE7BwR9xTVHynm++PNP8uI2Csi7i7mdnZEHNVszBMj4vai3wci4t0tza9UWpnpy5evbvYC5gEfaKfOfcDJ7dT5LPAr4G1AL2A0sHWx7nbgRmA7oA9wUFF+CPASMArYArgcuKeqzwT+B9ge2BIYCfwV2LfYxqeK8W/RwnjeD8wHoni/HbAC2JHK/5GcAXwD2BzYDXgWGFfUPR9YBRxT1N0SuB/4ZLH+7cB+xfJYYEFrc9pauxbG21I/5wM/KZYHFvNxJdAX+CCVQH0r8E5gp2JuGuf2aGAusBfQG/g68IdWtt3Y9w3AVsBwYFHVPnwR+CMwoPicrgJuaNb2v4u2W7ayb6uB7xXtDwKWA4OL9ZOAl4EDi/l+Wzufz0XAvcX3Ymfgseq5azb/E4BHgcFAAHsD/aq+X+9p6TOg8j2dC/xrMYZDgFebjXkxMKaY358Ck7v6v2dfvjbmyyNuUvd1a3FUYWn10bD1tAroR+UP4ZrMnJGZr0TEu4APAWdk5pLMXJWZvyvanAhck5n/m5mvAecC+0fEwKp+/z0z/5aZK4DTgasy84FiGz8GXgP2a2E891L5w/y+4v2xwP1ZOQ28D9A/My/IzNcz81ngh8D4qvb3Z+atmflGse1VwHsiYofMXJaZf1yPeXkr7Vrzb5m5MjOnUQk/N2TmXzNzYbHPI4t6Z1CZuycyczXw/4ARrR11K3wrM5dn5qPAtcAJVX19LTMXFJ/T+cCxse5p0fOLtiva6P+8zHyt+PxvB46vWvfLzPx9Zr5BJTi29fkcD1xYfC/mA5e1sc1PA1/PzDlZ8UhmLm6jfqP9qATti4ox3AXcVjUnAL/IzAeL+f0pMKID/UqlYXCTuq9jMnPb4nVMRxrEujcz7AJcB0wFJhen274TEX2oHBH5W2YuaaGbHYE/Nb7JzGVUjmLsVFVnftXyrsC/VIXMpUX/O9JMZiYwmbV/aD9B5Y9rYz87NuvnX4F/aGW7AKcBewBPRuU08JGtzc1Gateav1Qtr2jh/duL5V2B/6zav79ROeJUPbfNVe/zn1g7r7sCv6jq6wlgDW3PV3NLMnN5K/03b9/e57NjC2Ntzc7AM+2MrSU7AvOLIFm9ner5+3PV8t9ZO/fSJsGLe6VNSGa29EfqW8C3iiNmdwBzin+3j4htM3Nps/ovUPkjDTRd69UPWFi9qarl+VSOtFzYwWHeAEyLiIuonF79SFU/z2Xm7m20zXXeZD4NnBARmwEfBW6OiH5Ujnq9rWofegH922vXLMS8aXsbQeNc/bTdmmvtDDxZLO9C5fNp7OvUzPx98wZVR0fbG/92EbFV1X7vQuUUZ6Pmn3Nbn8+LxVhnV/XVmvnAu5ttqyNeAHaOiM2qwtsuwFPr2Y9UWh5xk0omIjaPiL5UjtT0icoNCC3+txwRB0fE8CK4vELlFOEbmfkilQvvr4iI7SKiT0S8v2h2A3BKRIyIiC2onM57IDPntTKkHwJnRMS+UbFVRBwREe9oqXJmzqRyDd2PgKlVwfFB4NWI+GpxoXuviBgWEfu01E+xfydFRP/ij3hjP29Q+UPetxhHHyrXkm3RgXbN/QXoFxHbtDaG9XQlcG5EDC3GsU1EHNdOm/Mi4m1Fm1OoXJfY2NeFjadZI6J/RBz9Fsb0reI79T7gSOBnrdRr7/O5qdi37SJiAPCFNrb5I+DfImL34jtTVwRuqMz5bq20e4DKUbSvFN/ZscA/UjmKK/UIBjepfKZROf12AHB1sfz+Vur+H+BmKqHtCeB3VE6fAnySSpB7ksoF9P8XIDN/C5wH3ELlKMq7Wfc6s3VkZgPwGeD7wBIqF4+f3M4+XA98oPi3sZ81VILDCOA51oa7tkLT4cDsiFgG/CcwPjNXZObLVH4y5UdUjhQuBxa0166FfXuSSpB9tjg9+KbTv+sjM38BXEzl1PUrVI44faidZr+jMqd3At8trqOjGPcUKkcvX6Vyo8K+6zmkP1P5zF6gcsr6jGKfWxp7e5/Pt6ictnyOynf0uha6afQ9KkFvGpXv5n9RudkEKtfq/biY7+rr7cjM16kEtQ8V278C+KfWxixtihrv7JIkdSPF6c7ngD7FhfYbu/+xVO6OHbCx+5ZUOx5xkyRJKgmDmyRJUkl4qlSSJKkkPOImSZJUEj3id9x22GGHHDhwYFcPQ5IkqV0zZsx4KTP7t7SuRwS3gQMH0tDQ0NXDkCRJaldEtPrkEU+VSpIklYTBTZIkqSQMbpIkSSXRI65xa8mqVatYsGABK1eu7OqhqIfr27cvAwYMoE+fPl09FElSN9djg9uCBQt4xzvewcCBA4mIrh6OeqjMZPHixSxYsIBBgwZ19XAkSd1cjz1VunLlSvr162doU5eKCPr16+eRX0lSh/TY4AYY2tQt+D2UJHVUjw5ukiRJZWJw60Lz5s1j2LBhNen77rvv5sgjjwRgypQpXHTRRTXZjiRJ6jw99uaEnuSoo47iqKOO6uphSJKkDVTTI24RcXhEzImIuRFxTgvrd4mI6RExMyJmRcSHi/J+RfmyiPh+szZ3F30+XLzeWct9aHTrzIUceNFdDDrndg686C5unblwo/S7evVqTjzxRPbaay+OPfZY/v73v3PBBRewzz77MGzYME4//XQyE4DLLruMIUOGUFdXx/jx4wFYvnw5p556KmPGjGHkyJH88pe/fNM2Jk2axFlnnQXAySefzNlnn80BBxzAbrvtxs0339xU75JLLmGfffahrq6Ob37zmxtl/yRJ0sZTs+AWEb2AicCHgCHACRExpFm1rwM3ZeZIYDxwRVG+EjgP+HIr3Z+YmSOK1183/ujXdevMhZz780dZuHQFCSxcuoJzf/7oRglvc+bM4cwzz+SJJ55g66235oorruCss87ioYce4rHHHmPFihXcdtttAFx00UXMnDmTWbNmceWVVwJw4YUXcsghh/Dggw8yffp0JkyYwPLly9vc5osvvsh9993HbbfdxjnnVPL0tGnTePrpp3nwwQd5+OGHmTFjBvfcc88G758kSdp4annEbQwwNzOfzczXgcnA0c3qJLB1sbwN8AJAZi7PzPuoBLgud8nUOaxYtWadshWr1nDJ1Dkb3PfOO+/MgQceCMBJJ53Efffdx/Tp09l3330ZPnw4d911F7Nnzwagrq6OE088kZ/85Cf07l05yz1t2jQuuugiRowYwdixY1m5ciXPP/98m9s85phj2GyzzRgyZAh/+ctfmvqZNm0aI0eOZNSoUTz55JM8/fTTG7x/kiRp46nlNW47AfOr3i8A9m1W53xgWkR8AdgK+EAH+742ItYAtwDfzsZziVUi4nTgdIBddtll/UbezAtLV6xX+fpo/lMQEcGZZ55JQ0MDO++8M+eff37Tb3zdfvvt3HPPPfzqV7/iwgsv5NFHHyUzueWWWxg8ePA6/TQGspZsscUWTcuNU5eZnHvuuXz2s5/d4H2SJGmTMusmuPMCeHkBbDMADv0G1B3fJUPp6rtKTwAmZeYA4MPAdRHR3phOzMzhwPuK1ydbqpSZV2dmfWbW9+/ff4MGueO2W65X+fp4/vnnuf/++wG4/vrree973wvADjvswLJly5quQXvjjTeYP38+Bx98MBdffDEvv/wyy5YtY9y4cVx++eVNAWzmzJlvaRzjxo3jmmuuYdmyZQAsXLiQv/615mehJUnq3mbdBL86G16eD2Tl31+dXSnvArUMbguBnaveDyjKqp0G3ASQmfcDfYEd2uo0MxcW/74KXE/llGxNTRg3mC379FqnbMs+vZgwbnArLTpu8ODBTJw4kb322oslS5bwuc99js985jMMGzaMcePGsc8++wCwZs0aTjrpJIYPH87IkSM5++yz2XbbbTnvvPNYtWoVdXV1DB06lPPOO+8tjeODH/wgn/jEJ9h///0ZPnw4xx57LK+++uoG758kSaV25wWwqtkZtlUrKuVdIFo4y7hxOo7oDTwFHEolsD0EfCIzZ1fV+TVwY2ZOioi9gDuBnRpPfUbEyUB9Zp5V1ee2mflSRPQBbgB+m5lXtjWW+vr6bGhoWKfsiSeeYK+99urw/tw6cyGXTJ3DC0tXsOO2WzJh3GCOGblTh9tLbVnf76MkqZOcvy2VS/KbCzh/aU02GREzMrO+pXU1u8YtM1dHxFnAVKAXcE1mzo6IC4CGzJwC/Avww4j4EpVZObkqtM2jcuPC5hFxDPBB4E/A1CK09QJ+C/ywVvtQ7ZiROxnUJEnqabYZUJwmbaG8C9T0B3gz8w7gjmZl36hafhw4sJW2A1vpdvTGGp8kSVKbDv1G5Zq26tOlfbaslHeBrr45QZIkqfuqOx7+8TLYZmcgKv/+42Vddlepj7ySJElqS93xXRbUmvOImyRJUkkY3CRJkkrC4CZJklQSBrcusnTpUq644oqm9xMmTGDo0KFMmDChxfonn3xy01MUOmrgwIG89NJLGzTO9fUf//Ef/P3vf+/UbXalu+++myOPPLKrhyFJ6iEMbh016ya4dFjlh/guHbbBj7poHtyuvvpqZs2axSWXXLKhI+1SPS24ra/Vq1d39RAkSSVmcOuIGjyn7JxzzuGZZ55hxIgRHHbYYSxbtozRo0dz4403ttrmnnvu4YADDmC33XZrOvrW/IjPWWedxaRJk5ref+c732H48OGMGTOGuXPnttr3z372M4YNG8bee+/N+9//fqDymK0JEyawzz77UFdXx1VXXdW0zbFjx3Lsscey5557cuKJJ5KZXHbZZbzwwgscfPDBHHzwwQBMmzaN/fffn1GjRnHcccc1PQt14MCBfPOb32TUqFEMHz6cJ598EoBly5ZxyimnMHz4cOrq6rjlllva7KclM2bM4KCDDmL06NGMGzeOF198EYCxY8fy1a9+lTFjxrDHHntw7733Nu3nl7/8ZYYNG0ZdXR2XX345AHfeeScjR45k+PDhnHrqqbz22msA/OY3v2HPPfdk1KhR/PznP2/a7vLlyzn11FMZM2YMI0eO5Je//CUAkyZN4qijjuKQQw7h0EMPbXXckiS1KzM3+dfo0aOzuccff/xNZa363tDMb2795tf3hna8j2aee+65HDp0bfutttqqzfqf+tSn8thjj801a9bk7Nmz893vfndmZk6fPj2POOKIpnqf//zn89prr83MzF133TW//e1vZ2bmj3/843XqNTds2LBcsGBBZmYuWbIkMzOvuuqq/Ld/+7fMzFy5cmWOHj06n3322Zw+fXpuvfXWOX/+/FyzZk3ut99+ee+99zZtc9GiRZmZuWjRonzf+96Xy5Yty8zMiy66KL/1rW811bvssssyM3PixIl52mmnZWbmV77ylfziF7/YNK6//e1vbfbT3Ouvv577779//vWvf83MzMmTJ+cpp5ySmZkHHXRQ/vM//3NmZt5+++156KGHZmbmFVdckR/72Mdy1apVmZm5ePHiXLFiRQ4YMCDnzJmTmZmf/OQn89JLL20qf+qpp/KNN97I4447rmlezz333Lzuuuua5nD33XfPZcuW5bXXXps77bRTLl68uNX5X6/voyRpk0blCVMtZhp/x60jXl6wfuU1cswxx7DZZpsxZMgQ/vKXv3SozQknnND075e+9KVW6x144IGcfPLJHH/88Xz0ox8FKke5Zs2a1XR07+WXX+bpp59m8803Z8yYMQwYUHncx4gRI5g3bx7vfe971+nzj3/8I48//jgHHlh5OMbrr7/O/vvv37S+cTujR49uOnL129/+lsmTJzfV2W677bjtttva7KfanDlzeOyxxzjssMOAytG0d73rXS1uc968eU3bPOOMM+jdu/Kfw/bbb88jjzzCoEGD2GOPPQD41Kc+xcSJExk7diyDBg1i9913B+Ckk07i6quvbpqvKVOm8N3vfheAlStX8vzzzwNw2GGHsf3227c6/5IkdYTBrSO6yXPKtthii6blSiCH3r1788YbbzSVr1y5cp02EdHicnNXXnklDzzwALfffjujR49mxowZZCaXX34548aNW6fu3Xffvc5YevXq1eK1W5nJYYcdxg033NDm/rTWvqP9NK87dOhQ7r///g3a5luRmdxyyy0MHjx4nfIHHniArbbaaqNuS5LUM3mNW0cc+o3Kc8mqbeBzyt7xjnfw6quvbuDAYNddd+Xxxx/ntddeY+nSpdx5553rrG+8Zu7GG29s9SgVwDPPPMO+++7LBRdcQP/+/Zk/fz7jxo3jBz/4AatWrQLgqaeeYvny5W2Op3q/9ttvP37/+983XVu3fPlynnrqqTbbH3bYYUycOLHp/ZIlS9arn8GDB7No0aKm4LZq1Spmz57d7javuuqqpiD3t7/9jcGDBzNv3rymbV533XUcdNBB7LnnnsybN49nnnkGYJ0wOW7cOC6//PKmUD1z5sw2tytJ0voyuHVEDZ5T1q9fPw488ECGDRvW6k+AdMTOO+/M8ccfz7Bhwzj++OMZOXLkOuuXLFlCXV0d//mf/8mll17aaj8TJkxg+PDhDBs2jAMOOIC9996bT3/60wwZMoRRo0YxbNgwPvvZz7Z7lOr000/n8MMP5+CDD6Z///5MmjSJE044gbq6Ovbff/+mmxBa8/Wvf50lS5Y03Sgxffr09epn88035+abb+arX/0qe++9NyNGjOAPf/hDm9v89Kc/zS677EJdXR177703119/PX379uXaa6/luOOOY/jw4Wy22WacccYZ9O3bl6uvvpojjjiCUaNG8c53vrOpn/POO49Vq1ZRV1fH0KFDOe+889rcriRJ6ysajw5syurr67OhoWGdsieeeIK99tqri0YkrcvvoySpUUTMyMz6ltZ5xE2SJKkkvDmhm7nwwgv52c9+tk7Zcccdx9e+9rVS9N+ZPvKRj/Dcc8+tU3bxxRe/6WYKSZI2FT36VOmee+7Z5p2WUmfITJ588klPlUqSAE+Vtqhv374sXryYnhBc1X1lJosXL6Zv375dPRRJUgn02FOlAwYMYMGCBSxatKirh6Ierm/fvk0/ZixJUlt6bHDr06cPgwYN6uphSJIkdViPPVUqSZJUNgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKoqbBLSIOj4g5ETE3Is5pYf0uETE9ImZGxKyI+HBR3q8oXxYR32/WZnREPFr0eVlERC33QZIkqbuoWXCLiF7AROBDwBDghIgY0qza14GbMnMkMB64oihfCZwHfLmFrn8AfAbYvXgdvvFHL0mS1P3U8ojbGGBuZj6bma8Dk4Gjm9VJYOtieRvgBYDMXJ6Z91EJcE0i4l3A1pn5x8xM4L+BY2q4D5IkSd1GLYPbTsD8qvcLirJq5wMnRcQC4A7gCx3oc0E7fQIQEadHRENENCxatGh9xi1JktQtdfXNCScAkzJzAPBh4LqI2ChjysyrM7M+M+v79++/MbqUJEnqUrUMbguBnaveDyjKqp0G3ASQmfcDfYEd2ulzQDt9SpIkbZJqGdweAnaPiEERsTmVmw+mNKvzPHAoQETsRSW4tXpeMzNfBF6JiP2Ku0n/CfhlLQYvSZLU3fSuVceZuToizgKmAr2AazJzdkRcADRk5hTgX4AfRsSXqNyocHJx0wERMY/KjQubR8QxwAcz83HgTGASsCXw6+IlSZK0yYsiJ23S6uvrs6GhoauHIUmS1K6ImJGZ9S2t6+qbEyRJktRBBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVRE2DW0QcHhFzImJuRJzTwvpdImJ6RMyMiFkR8eGqdecW7eZExLiq8nkR8WhEPBwRDbUcvyRJUnfSu1YdR0QvYCJwGLAAeCgipmTm41XVvg7clJk/iIghwB3AwGJ5PDAU2BH4bUTskZlrinYHZ+ZLtRq7JElSd1TLI25jgLmZ+Wxmvg5MBo5uVieBrYvlbYAXiuWjgcmZ+VpmPgfMLfqTJEnqsWoZ3HYC5le9X1CUVTsfOCkiFlA52vaFDrRNYFpEzIiI01vbeEScHhENEdGwaNGit74XkiRJ3URX35xwAjApMwcAHwaui4j2xvTezBwFfAj4fES8v6VKmXl1ZtZnZn3//v037qglSZK6QC2D20Jg56r3A4qyaqcBNwFk5v1AX2CHttpmZuO/fwV+gadQJUlSD1HL4PYQsHtEDIqIzancbDClWZ3ngUMBImIvKsFtUVFvfERsERGDgN2BByNiq4h4R1F/K+CDwGM13AdJkqRuo2Z3lWbm6og4C5gK9AKuyczZEXEB0JCZU4B/AX4YEV+icu3ayZmZwOyIuAl4HFgNfD4z10TEPwC/iIjGsV+fmb+p1T5IkiR1J1HJSZu2+vr6bGjwJ98kSVL3FxEzMrO+pXVdfXOCJEmSOsjgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqiQ4Ft4jYIyLujIjHivd1EfH12g5NkiRJ1Tp6xO2HwLnAKoDMnAWMr9WgJEmS9GYdDW5vy8wHm5Wt3tiDkSRJUus6Gtxeioh3AwkQEccCL9ZsVJIkSXqT3h2s93ngamDPiFgIPAecWLNRSZIk6U3aDW4R0Qs4MzM/EBFbAZtl5qu1H5okSZKqtRvcMnNNRLy3WF5e+yFJkiSpJR09VTozIqYAPwOawltm/rwmo5IkSdKbdDS49QUWA4dUlSVgcJMkSeokHQpumXlKrQciSZKktnX0yQkDIuIXEfHX4nVLRAyo9eAkSZK0Vkd/x+1aYAqwY/H6VVEmSZKkTtLR4NY/M6/NzNXFaxLQv4bjkiRJUjMdDW6LI+KkiOhVvE6icrOCJEmSOklHg9upwPHAn6k86upYwBsWJEmSOlFH7yr9E3BUjcciSZKkNnT0rtIfR8S2Ve+3i4hrajcsSZIkNdfRU6V1mbm08U1mLgFG1mZIkiRJaklHg9tmEbFd45uI2J6OP3VBkiRJG0FHw9f/B9wfET8DgsrNCRfWbFSSJEl6k47enPDfEdHA2meVfjQzH6/dsCRJktRcR29OeDfwTGZ+H3gM+ED1zQpttDs8IuZExNyIOKeF9btExPSImBkRsyLiw1Xrzi3azYmIcR3tU5IkaVPV0WvcbgHWRMR7gKuAnYHr22oQEb2AicCHgCHACRExpFm1rwM3ZeZIYDxwRdF2SPF+KHA4cEXjj/92oE9JkqRNUkeD2xuZuRr4KPD9zJwAvKudNmOAuZn5bGa+DkwGjm5WJ4Gti+VtgBeK5aOByZn5WmY+B8wt+utIn5IkSZukjga3VRFxAvBPwG1FWZ922uwEzK96v6Aoq3Y+cFJELADuAL7QTtuO9AlARJweEQ0R0bBo0aJ2hipJktT9dTS4nQLsD1yYmc9FxCDguo2w/ROASYnBoj4AABOvSURBVJk5APgwcF1EdHRMbcrMqzOzPjPr+/fvvzG6lCRJ6lIdvav0ceBsgIgYlZn/C1zcTrOFVK6FazSgKKt2GpVr2MjM+yOiL7BDO23b61OSJGmT9FaObv2og/UeAnaPiEERsTmVmw2mNKvzPHAoQETsBfQFFhX1xkfEFsXRvd2BBzvYpyRJ0ibprTz9IDpSKTNXR8RZwFSgF3BNZs6OiAuAhsycAvwL8MOI+BKVGxVOzswEZkfETcDjwGrg85m5BqClPt/CPkiSJJVOVHLSejSIOCYzb63ReGqivr4+GxoaunoYkiRJ7YqIGZlZ39K69T5V2hjaImLPDR2YJEmSOm5D7uCcttFGIUmSpHa1eY1bRFzW2iqg3UdeSZIkaeNp7+aEU6jcQPBaC+tO2PjDkSRJUmvaC24PAY9l5h+ar4iI82syIkmSJLWoveB2LLCypRWZOWjjD0eSJEmtae/mhLdn5t87ZSSSJElqU3vBren32iLilhqPRZIkSW1oL7hVPyVht1oORJIkSW1rL7hlK8uSJEnqZO3dnLB3RLxC5cjblsUyxfvMzK1rOjpJkiQ1aTO4ZWavzhqIJEmS2rYhj7ySJElSJzK4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKoqbBLSIOj4g5ETE3Is5pYf2lEfFw8XoqIpZWrbs4Ih4rXh+vKp8UEc9VtRtRy32QJEnqLnrXquOI6AVMBA4DFgAPRcSUzHy8sU5mfqmq/heAkcXyEcAoYASwBXB3RPw6M18pqk/IzJtrNXZJkqTuqJZH3MYAczPz2cx8HZgMHN1G/ROAG4rlIcA9mbk6M5cDs4DDazhWSZKkbq+WwW0nYH7V+wVF2ZtExK7AIOCuougR4PCIeFtE7AAcDOxc1eTCiJhVnGrdopU+T4+IhohoWLRo0YbuiyRJUpfrLjcnjAduzsw1AJk5DbgD+AOVo3D3A2uKuucCewL7ANsDX22pw8y8OjPrM7O+f//+NR6+JElS7dUyuC1k3aNkA4qyloxn7WlSADLzwswckZmHAQE8VZS/mBWvAddSOSUrSZK0yatlcHsI2D0iBkXE5lTC2ZTmlSJiT2A7KkfVGst6RUS/YrkOqAOmFe/fVfwbwDHAYzXcB0mSpG6jZneVZubqiDgLmAr0Aq7JzNkRcQHQkJmNIW48MDkzs6p5H+DeSjbjFeCkzFxdrPtpRPSnchTuYeCMWu2DJElSdxLr5qVNU319fTY0NHT1MCRJktoVETMys76ldd3l5gRJkiS1w+AmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklURNg1tEHB4RcyJibkSc08L6SyPi4eL1VEQsrVp3cUQ8Vrw+XlU+KCIeKPq8MSI2r+U+SJIkdRc1C24R0QuYCHwIGAKcEBFDqutk5pcyc0RmjgAuB35etD0CGAWMAPYFvhwRWxfNLgYuzcz3AEuA02q1D5IkSd1JLY+4jQHmZuazmfk6MBk4uo36JwA3FMtDgHsyc3VmLgdmAYdHRACHADcX9X4MHFOT0UuSJHUztQxuOwHzq94vKMreJCJ2BQYBdxVFj1AJam+LiB2Ag4GdgX7A0sxc3YE+T4+IhohoWLRo0QbvjCRJUlfrLjcnjAduzsw1AJk5DbgD+AOVo3D3A2vWp8PMvDoz6zOzvn///ht7vJIkSZ2ulsFtIZWjZI0GFGUtGc/a06QAZOaFxfVvhwEBPAUsBraNiN4d6FOSJGmTUsvg9hCwe3EX6OZUwtmU5pUiYk9gOypH1RrLekVEv2K5DqgDpmVmAtOBY4uqnwJ+WcN9kCRJ6jZ6t1/lrcnM1RFxFjAV6AVck5mzI+ICoCEzG0PceGByEcoa9QHurdyLwCvASVXXtX0VmBwR3wZmAv9Vq32QJEnqTmLdvLRpqq+vz4aGhq4ehiRJUrsiYkZm1re0rrvcnCBJkqR2GNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklUTvrh5A2d06cyGXTJ3DC0tXsOO2WzJh3GCOGblTVw9LkiRtggxuG+DWmQs59+ePsmLVGgAWLl3BuT9/FMDwJkmSNjpPlW6AS6bOaQptjVasWsMlU+d00YgkSdKmzOC2AV5YumK9yiVJkjaEwW0D7LjtlutVLkmStCEMbhtgwrjBbNmn1zplW/bpxYRxg7toRJIkaVPmzQkboPEGBO8qlSRJncHgtoGOGbmTQU2SJHUKT5VKkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkqipsEtIg6PiDkRMTcizmlh/aUR8XDxeioillat+05EzI6IJyLisoiIovzuos/Gdu+s5T5IkiR1FzX7HbeI6AVMBA4DFgAPRcSUzHy8sU5mfqmq/heAkcXyAcCBQF2x+j7gIODu4v2JmdlQq7FLkiR1R7U84jYGmJuZz2bm68Bk4Og26p8A3FAsJ9AX2BzYAugD/KWGY5UkSer2ahncdgLmV71fUJS9SUTsCgwC7gLIzPuB6cCLxWtqZj5R1eTa4jTpeY2nUFvo8/SIaIiIhkWLFm343kiSJHWx7nJzwnjg5sxcAxAR7wH2AgZQCXuHRMT7ironZuZw4H3F65MtdZiZV2dmfWbW9+/fv+Y7IEmSVGu1DG4LgZ2r3g8oyloynrWnSQE+AvwxM5dl5jLg18D+AJm5sPj3VeB6KqdkJUmSNnm1fMj8Q8DuETGISmAbD3yieaWI2BPYDri/qvh54DMR8e9AULkx4T8iojewbWa+FBF9gCOB37Y3kBkzZrwUEX/a0B3qBnYAXurqQXQDzsNazsVazsVazkWF87CWc7FWGeZi19ZW1Cy4ZebqiDgLmAr0Aq7JzNkRcQHQkJlTiqrjgcmZmVXNbwYOAR6lcqPCbzLzVxGxFTC1CG29qIS2H3ZgLJvEudKIaMjM+q4eR1dzHtZyLtZyLtZyLiqch7Wci7XKPhe1POJGZt4B3NGs7BvN3p/fQrs1wGdbKF8OjN64o5QkSSqH7nJzgiRJktphcCuXq7t6AN2E87CWc7GWc7GWc1HhPKzlXKxV6rmIdS8tkyRJUnflETdJkqSSMLhJkiSVhMGtxiLi8IiYExFzI+KcFtZvERE3FusfiIiBVevOLcrnRMS49vqMiEFFH3OLPjcvyk+OiEXFY8IejohP13avW9bJc3FWUZYRsUNVeUTEZcW6WRExqnZ73LJuMg9jI+Llqu/EOnd7d5ZOnoufFuWPRcQ1xc8KdYvvRFvjrlrfGXPRE78X/xURjxSf/c0R8fb2ttFZusk89Li/H1XrL4uIZR3ZRqfKTF81elH5rblngN2AzYFHgCHN6pwJXFksjwduLJaHFPW3oPIc12eK/lrtE7gJGF8sXwl8rlg+Gfh+D5uLkcBAYB6wQ9U2PkzlSRwB7Ac80EPnYSxwWw/7Tny4+NyDypNaPldV3mXfiW42Fz3xe7F1Vb/fA85paxs9cB5Opof9/Sja1QPXAcva20ZnvzziVltjgLmZ+Wxmvg5MBo5uVudo4MfF8s3AoRERRfnkzHwtM58D5hb9tdhn0eaQog+KPo+p4b6tr06bC4DMnJmZ81oYx9HAf2fFH4FtI+JdG3VP29Zd5qE76Oy5uKP43BN4kMpj+Bq30ZXfCdoad5XOmIvuoLPn4hWoHHkFtqTyo+9tbaOzdJd56A46dS4iohdwCfCVDm6jUxncamsnYH7V+wVFWYt1MnM18DLQr422rZX3A5YWfbS0rY9VHQKvfoZsZ+nMudjQcdRSd5kHgP2LUyO/joih67MTG0mXzEVxWvCTwG/WYxy11l3mAnrg9yIirgX+DOwJXN7ONjpLd5kH6Hl/P84CpmTmix3cRqcyuPUMvwIGZmYd8D+s/X8M6rn+F9g1M/em8j/Qt3bxeDrTFcA9mXlvVw+kG2g+Fz3ye5GZpwA7Ak8AH+/i4XSZVuahR/39iIgdgeNYN7h2Kwa32loIVP+/kwFFWYt1IqI3sA2wuI22rZUvpnKKp3ezcjJzcWa+VpT/iK55bFhnzsWGjqOWusU8ZOYrmbmsWL4D6BNVNy90kk6fi4j4JtAf+Of1HEetdYu56KnfC2h61OJk4GPtbKOzdIt56IF/P0YC7wHmRsQ84G0RMbedbXSuDb1IzlebF1T2Bp6lckFk48WPQ5vV+TzrXux4U7E8lHUvqHyWysWUrfYJ/Ix1b044s1h+V9X2PgL8cVOfi6o+57HuRflHsO6F6A/20Hn4P6z9Ae4xwPON7zfVuQA+DfwB2LLZNrr0O9HN5qJHfS+Kz/w9RdsAvgt8t61t9MB56LF/P4r2y9rbRqfPR1dstCe9qNy99RSVu1e+VpRdABxVLPelErjmUrlIeLeqtl8r2s0BPtRWn0X5bkUfc4s+tyjK/x2YXXwxpwN79oC5OJvKNQurgReAHxXlAUws6j8K1PfQeTir6jvxR+CAHvCdWF2UPVy8vtFdvhPdaC561PeCylmn3xef+2PATynurmxrGz1sHnrc349m260Obl3+nchMH3klSZJUFl7jJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCT1CUiol9EPFy8/hwRC4vlZRFxRVePrzNFxMCIeKxYro+Iy9qp/6/N3v+hluOT1H34cyCSulxEnE/l95K+29VjaUlE9M61zwHe6O0iYiBwW2YO62C/yzLz7es7Hknl5xE3Sd1KRIyNiNuK5fMj4scRcW9E/CkiPhoR34mIRyPiN8VD0omI0RHxu4iYERFTI+JdLfQ7KSKujIiGiHgqIo4syntFxCUR8VDxIO3PVo3j3oiYAjzeQn/LIuLSiJgdEXdGRP+i/O6I+I+IaAC+2NrYivJHIuIRKr/I3tL+vz0iri32d1ZEfCwiLgK2LI5O/rRxLMW/UezLY0Wbj1f1eXfxkPAnI+KnEREb6zOT1HkMbpK6u3cDhwBHAT8BpmfmcGAFcEQR3i4Hjs3M0cA1wIWt9DWQyqOcjgCujIi+wGnAy5m5D7AP8JmIGFTUHwV8MTP3aKGvrYCGzBwK/A74ZtW6zTOzHrisjbFdC3whKw90b815xdiGZ+Uh33dl5jnAiswckZknNqv/UWAEsDfwAeCSqhA7Evi/wBAqT1k5sI3tSuqmerdfRZK61K8zc1VEPErlGYO/KcofpRLEBgPDgP8pDiL1Al5spa+bMvMN4OmIeBbYE/ggUBcRxxZ1tgF2B16n8tzS51rp6w3gxmL5J8DPq9Y1lrc4tojYFtg2M+8p6l0HfKiFbXyAyjMRAcjMJa2MpdF7gRuy8qDwv0TE76iE0VeKfVkAEBEPU5m7+9rpT1I3Y3CT1N29BpCZb0TEqlx7Ye4bVP43LIDZmbl/B/pqflFvFu2/kJlTq1dExFhg+XqMs7rvxnYtjq0Ibp3ttarlNfi//1IpeapUUtnNAfpHxP4AEdEnIoa2Uve4iNgsIt5N5XThHGAq8Lmq6+X2iIitOrDdzYDGo3SfoOWjVy2OLTOXAksj4r1FveanPBv9D+te/7ZdsbiqcbzN3At8vLhurz/wfioPw5a0iTC4SSq1zHydSoC6uLjQ/2HggFaqP08lyPwaOCMzVwI/onLzwf8WP8lxFR07GrUcGFO0OQS4YD3HdgowsTht2dqNAt8GtituNngEOLgovxqY1XhzQpVfALOAR4C7gK9k5p87sC+SSsKfA5HUI0TEJCo/uXHzRurPn+SQ1Ok84iZJklQSHnGTJEkqCY+4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJ/P9ddOvt2ID1ewAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OTS1fYADMQs3"
      },
      "source": [
        "> 📖 **Resource:** See all course materials as well as exercises and extra-curriculum for this notebook on GitHub: https://github.com/mrdbourke/tensorflow-deep-learning"
      ]
    }
  ]
}